Page 1 of 1

Persistence for "Data Lists"

PostPosted:18 Nov 2013, 06:07
by gene
The discussion started in another thread:
gene wrote:During the last days I tried to develop a concept for saving our "data lists", so that we can advance to create a first "real" running prototype based on our data model (or at least a part of it). It would be great to use enumerations, but that's not easy. Look here or here. the latest (stable) hibernate version does not support JPA Version 2.1. I'll have to think about how we will deal with that.
Tracy Poff wrote:So I read about this and thought some, and I wanted to ask: is there some reason why you cannot (or should not) treat these as regular database entries, just like games or release groups? Certainly there are many of these that seem like a good fit for an enum (belonging to a limited set of choices, wanting to constrain values), but it looks like actually persisting enums in the database is a lot of trouble.

On the other hand, if these data lists are instead tables with entries for each item, there shall be no trouble storing them in the database (naturally), and we get the value constraint 'for free' as well. The program code can ensure that e.g. a Release Group links to some Release Group Reason, but it doesn't actually need to 'know' anything about what they are. Too, I think that we may want to treat some of them as objects in their own right, e.g. having descriptions and other metadata. Consider platforms or media types as examples of this.
Tracy Poff wrote:I was feeling a bit industrious, today, so I threw together a prototype of the server that treats ReleaseGroupTypes as database objects, rather than enums. The branch is here, and there's a corresponding branch of the client here. Of course, it's just a very simple change that I mainly did to get more familiar with the JPA. I followed the style of the existing code, more or less. It did bring up a couple of thoughts, though.
  • 1. If I did not do this in totally the wrong way, then there may be a bit of a mess, once everything is implemented--one DAO for each data list, plus one for each other type in the database. Is that right?

    2. If all of these data lists are made into entities in the DB, then the JSON that we return from the server may grow to be very large. Imagine that there is a lengthy description for each release group type (or each platform, or whatever), which is then duplicated in every single release group with that type (resp. platform, etc.). Is this likely to pose a problem, do you suppose? Though maybe the server can be made to emit partial representations of some of these, or something. Or, well, bytes are cheap, so maybe it just doesn't matter.
I suppose it shall be not too difficult to replicate this experiment for the other data lists, if you think that it's suitable. I welcome any feedback on my implementation, especially since this is my first time working with databases in java.

Re: Persistence for "Data Lists"

PostPosted:18 Nov 2013, 06:18
by gene
I had a look at your code changes for the persistence of "data lists" as own, real entities with ManyToOne-JPA-Relations.

I think this could work. In detail I thought about these aspects:
  • we should create a base class for data list types with a field "value" (that would replace your name)
  • every data list class extends from this base class and has no further own attributes but only the inherited ones
  • we should create a constructor with (Long key, String value) to create these central data list instances with a predefined key.
  • to fill in values we could the do this:
    Code: Select all
    	private void addReleaseGroupTypes() {		
    		rgtRepository.save(new ReleaseGroupType(ReleaseGroupType.ORIGINAL, "Original"));
    		rgtRepository.save(new ReleaseGroupType(ReleaseGroupType.ENHANCED, "Enhanced"));
    		rgtRepository.save(new ReleaseGroupType(ReleaseGroupType.DEMO, "Demo"));
    		rgtRepository.save(new ReleaseGroupType(ReleaseGroupType.REMAKE, "Remake"));
    	}
    with something like this in the data list class:
    Code: Select all
    	public final static Long ORIGINAL = 1L;
    	public final static Long ENHANCED = 2L;
    	public final static Long DEMO = 3L;
    	public final static Long REMAKE = 4L;
    This would assure that these values don't change over time (old data will stay consistent)
What do you think?

I also tried to create some JUnit tests with JPA persistence, but I did not succeed. We should try to accomplish this to test our code.

Re: Persistence for "Data Lists"

PostPosted:18 Nov 2013, 07:00
by Tracy Poff
gene wrote:we should create a base class for data list types with a field "value" (that would replace your name)
This is absolutely a good idea.
gene wrote:every data list class extends from this base class and has no further own attributes but only the inherited ones
I'm not totally sure about this. I think there may be specific cases that will require additional attributes. For example, it might be desirable to have some platforms be 'sub-platforms' of others, and the most natural way to implement this would be to put an extra attribute 'parent' on the platform type to allow this association. I use this example because I've been thinking about platforms recently, but maybe some other types will need such things also. Human roles in game development, for example.

However, for almost all cases, I agree with this.
gene wrote:we should create a constructor with (Long key, String value) to create these central data list instances with a predefined key.
to fill in values we could the do this:
Code: Select all
	private void addReleaseGroupTypes() {		
		rgtRepository.save(new ReleaseGroupType(ReleaseGroupType.ORIGINAL, "Original"));
		rgtRepository.save(new ReleaseGroupType(ReleaseGroupType.ENHANCED, "Enhanced"));
		rgtRepository.save(new ReleaseGroupType(ReleaseGroupType.DEMO, "Demo"));
		rgtRepository.save(new ReleaseGroupType(ReleaseGroupType.REMAKE, "Remake"));
	}
with something like this in the data list class:
Code: Select all
	public final static Long ORIGINAL = 1L;
	public final static Long ENHANCED = 2L;
	public final static Long DEMO = 3L;
	public final static Long REMAKE = 4L;
This would assure that these values don't change over time (old data will stay consistent)
Just to be clear, your only reason for this is to ensure that keys stay consistent? Because a major benefit of storing these in the database only is that no code changes are necessary on the server when we add new items to the data lists, and we'll lose that, if we do this. I think that the only time a key would change is if the database were wiped and things were re-inserted (i.e. not a database backup/restore), which is happening right now during development, but should never happen during production.

Could you expand a little more on this issue?
gene wrote:I also tried to create some JUnit tests with JPA persistence, but I did not succeed. We should try to accomplish this to test our code.
I've used unittest in python, which is based on JUnit, but I admit that I've always found it difficult to write useful unit tests on real code. For my database apps, I just gave up. Not a big problem, since they were for personal use, but not an option in this case--testing is necessary to ensure code quality. I'll add 'JUnit documentation' to my list of things to read, I guess.

Re: Persistence for "Data Lists"

PostPosted:19 Nov 2013, 00:51
by gene
I got JUnit tests working, see my committed "PersistenceTest" class.

I also committed a first try of better "Data List" classes, feel free to have a look at it.

Game.class:
Code: Select all
	@ManyToOne(cascade = CascadeType.ALL)
	private GameEntryType gameEntryType;
This seems to work in my unit test.

Need to go to bed now...

Re: Persistence for "Data Lists"

PostPosted:26 Nov 2013, 07:41
by gene
I got some more persistence tests working!

I added a constructor to the BaseDataList class:
Code: Select all
	public BaseDataList(String value) {
		super();
		this.value = value;
	}
and removed the setter for the "value" field.

Besides I changed the CascadeType for the field Game.gameEntryType from "ALL" to "none" (just deleted the ALL).
Only this way if a Game is deleted the associated GameEntryType stays in the database.

This pattern should work for all DataLists, so we can now proceed to fill in all DataList fields from our data model.

Re: Persistence for "Data Lists"

PostPosted:27 Nov 2013, 00:25
by Tracy Poff
I'd like to reiterate my question regarding the use of hardcoded constants as values in the database:
Tracy Poff wrote:Just to be clear, your only reason for this is to ensure that keys stay consistent? Because a major benefit of storing these in the database only is that no code changes are necessary on the server when we add new items to the data lists, and we'll lose that, if we do this. I think that the only time a key would change is if the database were wiped and things were re-inserted (i.e. not a database backup/restore), which is happening right now during development, but should never happen during production.

Could you expand a little more on this issue?
Once again, if we do this, it will mean that adding new platforms, currencies, goodie types, human/company roles, or any other data list item will require a change in the code. Since these items are essentially data, I'd prefer that they not be part of the program code, if possible.

Could you please explain how the benefits of using constants this way will outweigh this drawback?
gene wrote:Besides I changed the CascadeType for the field Game.gameEntryType from "ALL" to "none" (just deleted the ALL).
Only this way if a Game is deleted the associated GameEntryType stays in the database.
I was going to comment on this, some days ago, but when I tested it, deletes didn't seem to be propagating from Games to GameEntryTypes--I speculated that this was because of the ManyToOne relationship. Were you seeing Game deletion removing the associated GameEntryType from the database? Because I can't think of any reason we would have different results. It should definitely be consistent.
gene wrote:This pattern should work for all DataLists, so we can now proceed to fill in all DataList fields from our data model.
Once my query, above, is resolved, I'll be happy to dive into adding some of these. I'm excited to see the data model more fully implemented!

Although of course the data lists can be added independently of the rest of the data model, do you think that we should try to go through the whole data model and figure out what's 'final' and what still needs further discussion? It'd be good to review everything and see how mature our data model is, at the moment, I think.

Re: Persistence for "Data Lists"

PostPosted:27 Nov 2013, 22:33
by MZ per X
Tracy Poff wrote:Although of course the data lists can be added independently of the rest of the data model, do you think that we should try to go through the whole data model and figure out what's 'final' and what still needs further discussion? It'd be good to review everything and see how mature our data model is, at the moment, I think.
I'd rather go a different way, and use development sprints like gene suggested some time ago. Discussing the whole data model to the bone is not worthwhile, since some parts are still missing, and the real detailed questions come up in development only.

In a development sprint, we could specify the exact features we want to implement next, finalize the theoretical data model for these, draft the UI, then code and test it. This way, we could work on digestible pieces, and see results in reasonable time.

Re: Persistence for "Data Lists"

PostPosted:27 Nov 2013, 23:03
by Tracy Poff
MZ per X wrote:I'd rather go a different way, and use development sprints like gene suggested some time ago. Discussing the whole data model to the bone is not worthwhile, since some parts are still missing, and the real detailed questions come up in development only.

In a development sprint, we could specify the exact features we want to implement next, finalize the theoretical data model for these, draft the UI, then code and test it. This way, we could work on digestible pieces, and see results in reasonable time.
Sure, that's not so different from what I intended, anyway. I figured we'd take a quick look at the whole data model, sorting it broadly by 'probably ready' and 'probably not ready', pick the parts that seemed most mature, and then make some post saying "unless someone has important objections/suggestions, we're implementing these". It's definitely too soon to try to closely analyze the whole thing.

Re: Persistence for "Data Lists"

PostPosted:27 Nov 2013, 23:08
by gene
Tracy Poff wrote:Could you please explain how the benefits of using constants this way will outweigh this drawback?
Well that's not a big deal.
During development I would like to have some date of games with which I can "work" (mainly display, later edit). The way I chose to satisfy this need is the class "DatabaseFiller", which (how surprising) fills the database with some useful data. For this I do need some "reference" to reference the single values of the DataLists. We may alternatively choose the text, but that basically would have the same drawbacks.

If at some point in time when we have decided how we can fill these data list entries into the database for production (maybe with SQL scripts) we may delete the constants.
On the other hand if we may want to distinguish betwenn concrete values of a data list inside the code we do need some reference. And as the key of a productive data list entry will stay constant, why should we not write them down in the code to be used there? I am used to this pattern from different projects at work.

Yes, with "CascadeType.ALL" if I deleted a Game object in a unit test, the referenced GameEntry was also deleted. You may test this locally.

Well, what now (if we decided to leave things as they are right now)?
One pattern that is "open" are "tag lists", that would be "ManyToMany" relation. But I think that the "tags lists" are very similar to "data lists".

We could then try to build a basic part of the data model and then think about other things (authentication e.g.).

Re: Persistence for "Data Lists"

PostPosted:28 Nov 2013, 01:58
by Tracy Poff
gene wrote:During development I would like to have some date of games with which I can "work" (mainly display, later edit). The way I chose to satisfy this need is the class "DatabaseFiller", which (how surprising) fills the database with some useful data. For this I do need some "reference" to reference the single values of the DataLists. We may alternatively choose the text, but that basically would have the same drawbacks.
Yeah, I've of course modified this class in testing, too. If you wanted to totally avoid these kind of constants, then you could of course just assign the datalist values used in testing to variables and then deal with them that way, but if it's basically just for using while in development, it's not a problem.
gene wrote:On the other hand if we may want to distinguish betwenn concrete values of a data list inside the code we do need some reference. And as the key of a productive data list entry will stay constant, why should we not write them down in the code to be used there? I am used to this pattern from different projects at work.
When I suggested avoiding using constants, it was because I imagined that we should very probably never need to use some particular one of these in the code. If the client wants some filtering or other done with one, it can very well send that object's id along with its request. If we're wanting to do something with some particular data list values that the client won't know to send (say some filtering because a keep case has a spine but a paper slipcase does not), then that may be a case where our data model needs to be expanded to include that information. But given the choice, for sure I agree we'd do better to use the defined constants than just the text strings or whatever.

On the other hand, it seemed to me that if the normal operation of the program (and I think that adding new platforms and roles and things probably qualifies as 'normal operation') requires that the program be recompiled, it may be a sign of a flawed design.
gene wrote:Yes, with "CascadeType.ALL" if I deleted a Game object in a unit test, the referenced GameEntry was also deleted. You may test this locally.
It's so strange that when I tested that, the delete didn't propagate. I imagine I must have made some mistake in my test. I'll check again, later, but I'm sure you're right.
gene wrote:Well, what now (if we decided to leave things as they are right now)?
One pattern that is "open" are "tag lists", that would be "ManyToMany" relation. But I think that the "tags lists" are very similar to "data lists".
It looks to me like there's no essentially difference between the data lists and tag lists, except I guess that we intend relations with tag lists to be ManyToMany. In that case, we might well implement them as extending BaseDataList, or I guess we could have both BaseDataList and BaseTagList extend some BaseSomethingList, if we want to reflect the distinction in the class hierarchy.
gene wrote:We could then try to build a basic part of the data model and then think about other things (authentication e.g.).
Frankly, after writing those novels, I won't mind a vacation from authentication. And I think that if we're doing things 'in order', then before even authentication we probably need to talk about how the contribution system will work, like how we're storing unapplied changes in the database, and such.

Re: Persistence for "Data Lists"

PostPosted:28 Nov 2013, 14:38
by Ultyzarus
Tracy Poff wrote: If we're wanting to do something with some particular data list values that the client won't know to send (say some filtering because a keep case has a spine but a paper slipcase does not), then that may be a case where our data model needs to be expanded to include that information. But given the choice, for sure I agree we'd do better to use the defined constants than just the text strings or whatever.
With the cover templates I'm making (http://wiki.oregami.org/display/OR/Cove ... +Templates) this shouldn't be a problem, as each type of cover and variation will be documented. When I'm through doing this, I believe we expand Data Lists 17 & 18 with all this info.

Re: Persistence for "Data Lists"

PostPosted:16 Dec 2013, 22:41
by gene
I did some more extensions for the persistence of "data lists" and "tag lists".
I added:
  • GameEntryType
  • ReleaseType
  • ReleaseGroupReason
  • DemoContentType
  • and some more
The class "BaseList" is the main class all types extend.

The values for these types are filled into the database with the method "org.oregami.data.DatabaseFiller.initBaseLists()" (only for test/development, as mentioned above).
I also updated the client code a bit to reflect those changes on the demo website. It's only implemented prototypically, so dont's expect too much. It's not looking nice, but it helps us a lot to determine the technical patterns we will use for our objects.

Through the class "BaseListfinder" these type objects can be retrieved from the database.

I am not fully satisfied with the implementations. Disadvantages / problems to solve:
  • every type needs an own implementation of the DAO class
  • the method "findByName" is implemented in every DAO
  • and some more details. But it's an improvement!
I found the pattern of "AssistedInject" for google guice, but did not get it to work with the first try for the BaseList-DAOs. Need some more time for this to save duplicated code!

Edit: I fixed the URL of the demo site to http://test.client.oregami.org/#/games

Edit2: The actual JSON representation of a game looks like this:
Code: Select all
    {
        "id": 1,
        "version": 0,
        "lastUpdate": null,
        "gameEntryType": {
            "id": 6,
            "version": 0,
            "lastUpdate": null,
            "value": "GAME"
        },
        "gameTitleList": [
            {
                "id": 3,
                "version": 0,
                "lastUpdate": null,
                "title": "The Secret of Monkey Island"
            },
            {
                "id": 1,
                "version": 0,
                "lastUpdate": null,
                "title": "Monkey Island 1"
            },
            {
                "id": 2,
                "version": 0,
                "lastUpdate": null,
                "title": "Monkey Island"
            }
        ],
        "releaseGroupList": [
            {
                "id": 1,
                "version": 1,
                "lastUpdate": null,
                "name": "DOS-Demo",
                "releaseGroupReason": null,
                "releaseType": {
                    "id": 1,
                    "version": 0,
                    "lastUpdate": null,
                    "value": "NATIVE_DEVELOPMENT"
                },
                "demoContentTypeList": [
                    {
                        "id": 3,
                        "version": 0,
                        "lastUpdate": null,
                        "value": "CONTENT_LIMIT"
                    },
                    {
                        "id": 7,
                        "version": 0,
                        "lastUpdate": null,
                        "value": "TECH_DEMO"
                    }
                ],
                "censored": false,
                "released": true,
                "system": "MSDOS",
                "unreleaseState": null,
                "releaseList": []
            }
        ],
        "mainTitle": "The Secret of Monkey Island"
    }
(the part for game titles is not finished yet)