Developer's diary

[gene]

As you all know, we are unfortunately "short of developers".
I have been working on the basic system architecture for quiet some time now. It's fun for me, even though I don't advance as fast as I wish to. But I don't want to hurry, because if the basics are done badly, the whole system will become suboptimal.

This thread is meant to contain my "developers diary": I will try to post here regularly about my progress (which will not always be new source code, but often research results).

Feel free to watch this thread ond/or to post questions that you might already have concerning our upcoming development. I would like to communicate with someone else about development details, otherwise I feel so alone! So please cheer me up!

And of course it would be great that if you now some willing developer that you give him/her a hint about Oregami!

[Tracy Poff]

Feel free to watch this thread ond/or to post questions that you might already have concerning our upcoming development. I would like to communicate with someone else about development details, otherwise I feel so alone! So please cheer me up!

Since no one else has taken you up on this, I'll have a go.

What is the current status of things? I had a look at the repo, but you haven't pushed any commits since February, so I imagine it must not be up to date. I'm not asking "when will it ship?", but for prospective developers, it'd be good to know what the code looks like now and where we are in terms of a minimum viable product.

Now, I'm no java developer--I used java in college, but I've been using other languages, since then, so I'm well out of practice. And I've developed some web apps for personal use, but I'm not familiar with the technologies to be used for Oregami. All that said, I'd be willing to spend some time at least looking at the code, if I knew that what I was looking at was what I might need to be working on.

So give us an update! I'm excited to see progress here. Maybe I'll even be able to lend a hand.

[gene]

Thanks for your interest!

Well, you hit the wrong Github repo. More later.

Let me start with our projected system architecture:

First i would like to emphasize that I invested months to decide how our system should look like. I bought books, studied web sites and watched youtube videos to learn about how our system should look like. So it is not a "quick shot".

Server
The server will be a REST conform application. I cannot explain here in detail what a REST based approach is, there are many books about this topic. Instead I would suggest that you take a look at these resources:

• Intro to rest (14 minutes video)
• What is REST anyway? (15 minutes video)
• A Brief Introduction to REST good article from Stefan Tilkov, which I met jesterday in a Java talk

As the decision for java has been made (by me) before, I looked around how REST based services can be implemented with Java. I ended at Dropwizard, a lightweight framework to build such REST based web applications with minimal effort. I recommend this great three parts long tutorial to get more into this topic. For persistence we are using JPA/Hibernate with a relational database (HSQLDB at the moment, MySQL or others are also possible), there are many tutorials, articles and books about these things. I refused to use the Spring framework (for different reasons), instead I am using Google Guice for dependency injection.

Client
The client will be a web application which uses JavaScript and HTML5 to call the REST services from the server to display and manipulate data.
I chose AngularJS, a relative new JavaScript framework for interactive web applications. I heard of it the first time about one year ago, and I have experimented with it quiet some time. I like it very much! For anybody who wants to learn more about AngularJS I recommend to watch this excellent 60 minutes video or this article.

These are some of the core concepts I chose. Now more about the current status.

Server and Client are two different Github repositories!
Take a look at the oregami github main page:

These tho repositories are the relevant ones. The others are outdated (as it is written in BIG LETTERS in the description of both repos).
As you can see the last commits are only a few days old.

A guide about how to check out the server repo and start the web application locally is here in our wiki. A guide for the client repo has to be done.

I have jenkins build jobs for both repos here.
The server with the latest code is running at http://test.server.oregami.org , while the latest client is running at http://test.client.oregami.org

During the last days I tried to develop a concept for saving our "data lists", so that we can advance to create a first "real" running prototype based on our data model (or at least a part of it). It would be great to use enumerations, but that's not easy. Look here or here. the latest (stable) hibernate version does not support JPA Version 2.1. I'll have to think about how we will deal with that.

A relative big other topic is Hypermedia. I haven't used/implemented anything with this concept, this has to be done. Watch this video to learn more about it.

That's a first "quick" answer. I hope that some of your questions are answered, as I said before feel free to ask more questions. If you really want to get into development we sould take some time to talk about what exactly you can do.

[Tracy Poff]

Thanks for the information. Before I start, here, I'd like to point out that the developers page points at the old, outdated repo, which caused my confusion.

I've looked through a few of the resources you provided, and I've started playing around with the code to get a feel for it. AngularJS looks very cool--I may adopt it for some of my own projects. My java is pretty rusty, so I hope you'll forgive me if I ask a few stupid questions, just now.

Just starting off, I'm trying to understand the flow of data from the server to the client. It looks like the '/games' view on the client is being fed by list() in GamesResource, which is in turn just querying the database for every Game. Somehow these results get serialized to JSON and returned to the client.

Problem: the results contain a 'mainTitle', and I can't see where it's coming from. In fact, I grepped the sources for the server, and that string doesn't appear. Where is this coming from?

Question: is there an issue tracker or such somewhere? The ones on github are empty. If not, is there a roadmap or something similar, describing the development plan?

I don't want to waste your time, so I'll stop with this, for the moment. I've got to do quite a bit more reading, before I'm at all comfortable with this stuff. I just wanted to check in and confirm to you that I'll try to get up to speed. I hope that I may be able to aid in development, in the future.

[gene]

Well, first of all: Sorry for the confusion with the wrong link. I corrected it and added single links for server and client repository. This should make it clear.

Yeah, you got a lot of things right.
The #/games on the client makes a GET request to the Server, which ends in the GamesResource.list() method. This returns a Java list as JSON (Dropwizard does the conversion). The server URL that is called is http://test.server.oregami.org/games . AngularJS is used to iterate over the list of Game objects.
If you call the URL http://test.server.oregami.org/games it will lead to this method in GamesResource:

Code: Select all

@GET
@Path("/{id}")
	public Response getGame(@PathParam("id") long id) {

I would recomment that you use something like this Google Chrome plugin to try out the REST URLs, it's very comfortable to use and formats the JSON response.

The "mainTitle" comes from this method in the "Game" class:

Code: Select all

public String getMainTitle() {

Dropwizard uses the Jackson JSON library, read more about it here.

As issue tracker we will use our Jira installation. You have to register here.
We have different sub-projects, the ine for developers is / will be the Oregami WebApplication.
There is no detailed plan yet. We can use Jira to collect our tasks and sometimes later also bugs. I have also prepared a "Feature" item. We can talk about what we can do with Jira, the sky is the limit

[Tracy Poff]

If you call the URL http://test.server.oregami.org/games it will lead to this method in GamesResource:

Code: Select all

@GET
@Path("/{id}")
	public Response getGame(@PathParam("id") long id) {

Rather, that gets called if I request something like http://test.server.oregami.org/games/1.

The "mainTitle" comes from this method in the "Game" class:

Code: Select all

public String getMainTitle() {

I guessed that this had something to do with it, but I don't know what is causing this function to get bound to mainTitle in the JSON. It also doesn't behave like I think it should. If I remove the title from a game, then the mainTitle in the json is "[missing title for game with id 2!]", as expected. But, if a title does exist, it looks to me like this function should be returning something like "mt: Resident Evil", while the JSON will have simply "mainTitle":"Resident Evil", not "mainTitle":"mt: Resident Evil". Can you point out to me why this functions as it does?

As issue tracker we will use our Jira installation. You have to register here.
We have different sub-projects, the ine for developers is / will be the Oregami WebApplication.
There is no detailed plan yet. We can use Jira to collect our tasks and sometimes later also bugs. I have also prepared a "Feature" item. We can talk about what we can do with Jira, the sky is the limit

I had thought there was a jira instance, but when I looked again, I couldn't find it. I see now that it's linked from the community page, but not the developer page.

I recognize that the code is in an earlier-than-preliminary state just now, so I won't bug you more about how it's operating at the moment, without good reason. On to a more interesting topic.

During the last days I tried to develop a concept for saving our "data lists", so that we can advance to create a first "real" running prototype based on our data model (or at least a part of it). It would be great to use enumerations, but that's not easy. Look here or here. the latest (stable) hibernate version does not support JPA Version 2.1. I'll have to think about how we will deal with that.

(Warning in advance: my experience with databases is almost entirely with python+sqlalchemy, so maybe I don't know at all what I'm talking about.)

So I read about this and thought some, and I wanted to ask: is there some reason why you cannot (or should not) treat these as regular database entries, just like games or release groups? Certainly there are many of these that seem like a good fit for an enum (belonging to a limited set of choices, wanting to constrain values), but it looks like actually persisting enums in the database is a lot of trouble.

On the other hand, if these data lists are instead tables with entries for each item, there shall be no trouble storing them in the database (naturally), and we get the value constraint 'for free' as well. The program code can ensure that e.g. a Release Group links to some Release Group Reason, but it doesn't actually need to 'know' anything about what they are. Too, I think that we may want to treat some of them as objects in their own right, e.g. having descriptions and other metadata. Consider platforms or media types as examples of this.

[gene]

Yeah, of course I meant http://test.server.oregami.org/games/1 , sorry for the typo...

Concerning the "mainTitle":
This is absolutely not complete, it's just experimenting around.
As you noticed, the current code has to be considered as a technical prototype, and not (yet) a subject-specific.
I wish this would be more advanced, but the technical definitions were more important to me in the beginning.

So the current state is that I want to clarify the technical questions and then go an as soon as possible to build a subject-specifiv prototype, that means that we tre to implement our UML model step by step.

Thanks for your interesting tip about how to handle the persistence of our "data lists". Seems to be a good possibility to model them as regular entities. My thought was that we did not need them as seperate entites, but as you said if we want to build "meta information" about them it is absolutely necessary. I will have a try for this.

If we point out how we want to use Jira, we can of course link it from the developer page.

So much for now, it's early in the morning here...

[Tracy Poff]

Replying slightly out of order...

Thanks for your interesting tip about how to handle the persistence of our "data lists". Seems to be a good possibility to model them as regular entities. My thought was that we did not need them as seperate entites, but as you said if we want to build "meta information" about them it is absolutely necessary. I will have a try for this.

To expand upon this slightly: it seems to me that regardless of the difficulties with persisting enums, or the desire to otherwise treat them as objects, it would be a good idea to do it this way, in the interest of keeping the separation of code from data. It should also likely make it cleaner to update in the future, e.g. as new platforms are produced. The only drawback I can think of is that it expands the number of tables and connections between tables, making the underlying database structure a little more complicated, but I guess that is not too big a concern.

So the current state is that I want to clarify the technical questions and then go an as soon as possible to build a subject-specifiv prototype, that means that we tre to implement our UML model step by step.

In the spirit of clarifying, then, please allow me to 'think out loud' about a few of the major requirements for the server.

• 1. User authentication. It looks like dropwizard might handle this, but it's not clear to me if it's suitable. The description in the manual is rather sparse.
  
  2. Versioning of the data. This is important--we'll need at least the ability to rollback destructive changes (even with trusted users, accidents can happen, after all) more gracefully than by restoring a database backup. I have absolutely no idea how to handle this. Perhaps there's some easy, standard way that I'm unaware of.
  
  3. Accepting contributions. If I have not misunderstood, Oregami is intended to have a moderated contribution process, like Mobygames? Then some such system must also be built.

Do you have plans already for dealing with these?

[Tracy Poff]

I was feeling a bit industrious, today, so I threw together a prototype of the server that treats ReleaseGroupTypes as database objects, rather than enums. The branch is here, and there's a corresponding branch of the client here. Of course, it's just a very simple change that I mainly did to get more familiar with the JPA. I followed the style of the existing code, more or less. It did bring up a couple of thoughts, though.

• 1. If I did not do this in totally the wrong way, then there may be a bit of a mess, once everything is implemented--one DAO for each data list, plus one for each other type in the database. Is that right?
  
  2. If all of these data lists are made into entities in the DB, then the JSON that we return from the server may grow to be very large. Imagine that there is a lengthy description for each release group type (or each platform, or whatever), which is then duplicated in every single release group with that type (resp. platform, etc.). Is this likely to pose a problem, do you suppose? Though maybe the server can be made to emit partial representations of some of these, or something. Or, well, bytes are cheap, so maybe it just doesn't matter.

I suppose it shall be not too difficult to replicate this experiment for the other data lists, if you think that it's suitable. I welcome any feedback on my implementation, especially since this is my first time working with databases in java.

[idrougge]

I have a question:
1) Will I, with my antiquated PPC Macs, be able to contribute?
2) Will I be able to contribute without the computer going on its knees to keep up with modern 500-thread javascripts?

[gene]

I have a question:
1) Will I, with my antiquated PPC Macs, be able to contribute?
2) Will I be able to contribute without the computer going on its knees to keep up with modern 500-thread javascripts?

1) Which Browser (version) runs on that computer?
We should try to support "older" browsers as well, of course. We will see this later. It's always a matter of "costs".

2) That we will use JavaScript does not necessarily mean that your computer will "freeze" on loading a page
So I think it's all a matter of "which browser versions are supported".
We should avoid things like this one:

[gene]

I was feeling a bit industrious, today, so I threw together a prototype of the server that treats ReleaseGroupTypes as database objects, rather than enums. The branch is here, and there's a corresponding branch of the client here.
...

Did you commit and push your changes?
Does not seem to be the case?

[Tracy Poff]

Did you commit and push your changes?
Does not seem to be the case?

I definitely did. I'm not sure what problem you're having.



I had even checked it from a logged-out session before making that post, since I don't normally use github.

Edit: just to be sure, I created a pull request for both the server and the client. Of course, I don't necessarily expect you to merge them, but this way it should be more convenient for you to find my modified code, I guess.

[gene]

I definitely did. I'm not sure what problem you're having.

Hmm, when I looked at it, it said "This branch is 0 commits ahead and 0 commits behind master".
Whatever, I will have a look at it!

[Tracy Poff]

Oh, yes, there was another thing I meant to say. Knew I was forgetting something...

I'm have not yet read up on Guice, so I don't know exactly how it works. I assume that I need to do something in OregamiGuiceModule.java. I could have just copied and modified what's there (so, something like bind(new TypeLiteral<GenericDAO<ReleaseGroupType, Long>>() {}).to(ReleaseGroupTypeDao.class);, I guess), but since I don't understand quite what that's doing, I left it out. The thing runs and seems to work normally, but I assume that it's going to keep building new database connections or something, as it is. Guice's documentation is next on my to-read list, so... anyway, I just wanted to mention this.