Proposal: Authentication and Separation of Concerns

[Tracy Poff]

I've been thinking about authentication and some of the feature requests, and I have a suggestion which I think will simplify development substantially. In short, I propose that the server part of Oregami should not be concerned with users at all. To make a change to the database, let the server require a secret key, and, ideally, that the request come from a whitelisted source over a secure connection. This key will be held by the client. Then, let all concerns with user rights be managed by the client.

Cons

• 1. Only the official client will have the key, so changes can be made only through that client.
  2. The client will require a database backend to keep track of users, so it cannot be only HTML+JS.

Pros

• 1. The server will be simpler.
  2. Much simpler.

Further Analysis

As for the first drawback, I think that while it is significant, it is not crippling. While many different clients will require read-only access to the data, I find it difficult to imagine that many clients will require write access. This design change would affect only clients that require write access to the data.

As for the second, I believe that the client will likely require a database anyway. Many useful features, like messaging between users, forums, user game lists, and more, are much better suited for a client than the server, and will certainly require a central version of the client with a persistent datastore.

The list of pros speaks for itself.

Seriously, though: the server should be primarily concerned with storing and retrieving data. All the user-facing issues really ought to be separate, anyway. So, my proposal is that not only the submission of changes, but also the entire approval process, be handled purely by the client.

There are some details that would need to be worked out, but I'd like to hear your opinion on this, before I spend too much more time thinking about small details.

[jotaroraido]

Well, I'm not a developer (and not much of a programmer), but my initial reaction is "then how does the database keep track of who makes what changes?"

Say I go and add Super Mario Bros. to the database. It has to sit in pending status before it's approved by another user. How will it know who to attribute the submission to? How will it know that a following approval isn't actually just me clicking back through my submission again? What if I need to pull it from pending and revise the release date before putting it back in the queue? I don't think it would be appropriate to handle this part on the client side -- after all, the submission details and sources are much more a part of the database itself than they are a part of the display and aggregation.

[Tracy Poff]

Well, I'm not a developer (and not much of a programmer), but my initial reaction is "then how does the database keep track of who makes what changes?"

Because the client will send that information to the database when the change is approved.

Say I go and add Super Mario Bros. to the database. It has to sit in pending status before it's approved by another user. How will it know who to attribute the submission to?

You will be logged into the client, and it will send that information to the server, along with all the other details.

How will it know that a following approval isn't actually just me clicking back through my submission again?

Because you're logged into the client. The submission, until it's approved, is always in the jurisdiction of the client, so it can keep track of that.

What if I need to pull it from pending and revise the release date before putting it back in the queue?

Then you pull it back and work on it. Since it's never yet been submitted to the database, there's no rollback work to do or anything. Once you submit the final version and it's approved, it is only then sent over to the database.

I don't think it would be appropriate to handle this part on the client side -- after all, the submission details and sources are much more a part of the database itself than they are a part of the display and aggregation.

The information about the sources and such will certainly be stored in the server's database. However, the server can consume data from whatever source--for example, by importing data from another database. All the details about moderated contributions and such are matters of how we pick what goes into the database. We could, in theory, use any process whatever for that, from completely open to requiring a unanimous vote from all users, and the server shouldn't need to care about how this all comes to pass.

[gene]

Hi Tracy,
thanks for your proposal.

What you are suggesting is a rather radical change of the (up to now) planned system architecture.
While that is (of course) not "forbidden", I have some concerns about your proposed changes. I will try to list them:

• I chose a REST system architecture for various reasons (scalability, generalization, simplicity, decoupling client and server). We might very well lose some or even all of these benefits if we follow your suggestion.
• You say that the client needs a database anyway. Why that? This doesn't sound good to me. If you begin to store information on the server *and* on the client (what if more than one client has write access?) things become really complicated. And data that you want to save on a client cannot be displayed or manipulated by other clients. IMHO that is a killer argument. You are speaking of "other clients with read only access": but then you have to say "sorry, information about x and y is not available for other clients than the official one". Sounds bad.
• What technology should be used to build your client? You say that it "cannot be only HTML5+JS". What then?
• I understand that if you begin to think in detail about REST authentication and our implementation of it, you might get a headache I do not have practical experiance with this, but what I read about REST & co convinced me that will be a very useful approach and we have to get through these headaches...
• What do you think is the reason that big players on the web (ebay, paypal, twitter, github, google, amazon, flickr) are using REST APIs? Because of the benefits they get through that.

As I said I am not an expert in this area who has developed tons of REST APIs in reality. But in the past year I did quiet some research and came to the conclusion, that a RESTful architecture is a good choice for us.

Do you have any example application that uses an architecture like the one you are suggesting? Might help me to understand your desires.

[jotaroraido]

The information about the sources and such will certainly be stored in the server's database. However, the server can consume data from whatever source--for example, by importing data from another database. All the details about moderated contributions and such are matters of how we pick what goes into the database. We could, in theory, use any process whatever for that, from completely open to requiring a unanimous vote from all users, and the server shouldn't need to care about how this all comes to pass.

As I understand it, keeping track of the who, what, when of submissions is a crucial part of the database itself -- it maintains the research integrity of the information to know who put it there, what their comments were, and when it was added or further revised, even if the data is accessed through a third-party interface.

It seems to me like designing it the way you propose is just asking for trouble. What if we wind up with a second client (perhaps an administration tool separate of the main client) that has write access to the database, and someone uses that to change something in an entry that causes a submission sitting in the other client to become invalid? It shouldn't be the server's responsibility to notify the client -- that would violate the REST architecture Gene proposes, and as you say, it wouldn't even know about the pending submissions -- but by the same token it would be highly inefficient to force the client to regularly check that all pending submissions are still valid.

Personally, I believe it would be safer and more appropriate to let the client be just that: a gateway into the system. All administrative and maintenance functions should be part of the server itself, with the client simply providing an interface to access them.

[Tracy Poff]

Based on your replies... either I really don't know what I'm talking about, or I've explained it really badly. I'll give it one more try, but if it doesn't seem any better to you after this, I'll assume the fault is with me. No harm done, at least.

I chose a REST system architecture for various reasons (scalability, generalization, simplicity, decoupling client and server). We might very well lose some or even all of these benefits if we follow your suggestion.

The server would still be RESTful. As I said, from the perspective of a client requesting data from the server, this design change would be invisible. The only modification I'm suggesting is to move authentication and the approval process to a separate piece of software from the server that is storing the game database.

You say that the client needs a database anyway. Why that? This doesn't sound good to me.

It seems to me that services which would be useful to provide, like private messaging between users or forums, will require some database backend. Moreover, these are extremely distinct from the game database, and I'm uncomfortable with the idea of trying to make the server handle every possible thing. It's generally good design to have each logical component handle exactly one function, and to tie those together with clear interfaces.

I simply propose that those services which seem, to me, to be distinct, be supplied by distinct pieces of software.

If you begin to store information on the server *and* on the client (what if more than one client has write access?) things become really complicated. And data that you want to save on a client cannot be displayed or manipulated by other clients. IMHO that is a killer argument.

As I said earlier, this modification "will certainly require a central version of the client with a persistent datastore." In other words, alongside the hosted official Oregami database server will be an official hosted Oregami client. Which I expect we'd have anyway: do you honestly expect people to download the client, install a web server, and then run their own instances? It's not reasonable.

You are speaking of "other clients with read only access": but then you have to say "sorry, information about x and y is not available for other clients than the official one". Sounds bad.

I do not imagine that any information would be unavailable to any client. Rather, exactly one feature would be restricted to the official client: modifying the database.

What technology should be used to build your client? You say that it "cannot be only HTML5+JS". What then?

I'd imagine that essentially the same technologies used for the server would be used for the client, for simplicity's sake. What I meant by this was that since some database would be required to store the user table and such, the client would require at least a SQL server and an interface to it--more than just the HTML+JS that is providing the user interface.

I understand that if you begin to think in detail about REST authentication and our implementation of it, you might get a headache I do not have practical experiance with this, but what I read about REST & co convinced me that will be a very useful approach and we have to get through these headaches...

And that is why I absolutely do not propose abandoning the RESTful design of the server. My intention was, in fact, to make it easier to make such a design work, by moving things that are not so good a fit for a RESTful design into a separate piece of software.

Do you have any example application that uses an architecture like the one you are suggesting? Might help me to understand your desires.

Of course the exact proposal I made is specific to our situation, but I don't think it's such a radical idea. For example, OpenID is precisely a system that allows authentication to be dealt with by a different logical unit than the software needing authenticated users (i.e. you can authenticate to Google, and it promises to some other site that it has verified your ID).

In fact, even the current design is essentially like this: the MySQL database itself has a password which the server knows. The server authenticates users and then makes changes to the database on their behalf. My suggestion is to move this one layer further out: the server has a password that the client knows, and the client authenticates users and passes changes to the server on their behalf.

I feel like I must have at some point written something horribly ambiguous in my first post, but I can't figure out what. It's quite bewildering and frustrating.

Well, if you still think I'm talking nonsense, just let me know so I can drop it.

[hydr0x]

Just a hint to clear up the confusion: You're misusing the term server. A server is one physical (or pseudo-physical) unit, but not limited to one application. So when you speak of splitting up things between a client and a server, you're using the wrong or at least misleading terminology. What you actually mean (at least I think so) is splitting up back-end and front-end or, to use REST terminology, web service and web service client. The misleading part about this is that both service and client can run on the same server. Conceptually, most existing web applications already have these two parts, you just don't see that from the outside. It's the extreme form as separating data and presentation.

[Tracy Poff]

Just a hint to clear up the confusion: You're misusing the term server. A server is one physical (or pseudo-physical) unit, but not limited to one application. So when you speak of splitting up things between a client and a server, you're using the wrong or at least misleading terminology. What you actually mean (at least I think so) is splitting up back-end and front-end or, to use REST terminology, web service and web service client. The misleading part about this is that both service and client can run on the same server. Conceptually, most existing web applications already have these two parts, you just don't see that from the outside. It's the extreme form as separating data and presentation.

I don't think that's it... we already do split data from presentation by separating them into two software packages: oregami-server and oregami-client. My suggestion was to move some responsibilities from the server package to the client package.

Or maybe that is the source of confusion, after all. I'll be optimistic!

[gene]

Yeah, that's it. You are speaking about the "client" and mean something completely different than I do.


I do understand (at least a little bit) that you feel a desire to seperate game data from user data. Game data should be accessible for anybody to read, even without authentication. Well, perhaps we need to use some "client api key" if our traffic gets to high. But for read only access (e.g. from an emulator software or a games collection management software) there does not need to be a "user" logged in.
But despite this I think that the data for the contribution process strongly belongs to the same part of the application where the game data lies. We want to save exactly who did what when and which parts of the data are "accepted", which comments are made for which submissions and so on. And user data is a part of this. So I do not want to develop with more than one database (you proposed seperate databases for game data and user data), we would have to connect them somehow.

Besides I do not want to prevent any client from calling special REST urls/methods. Everybody who follows our (yet to be defined) rules can call every REST service. Why not have someone else develop an Android app to make Oregami approvals? As said before, we might demand an API key like ebay does ("Keys are unique identifiers that tell eBay which developer and application is making a call."). There might also be some test suite every "unofficial" client application must run through without errors.

Oh yes, of course nobody needs to install software to use "our" web client - it will be available on oregami.org at some time. The "server" part will be available publicly e.g. at api.oregami.org (take a look at this).


These "decisions" (if we come to an agreement) do not mean that developing everything will be easy. But they will in my opinion make our application robust and long-term maintainable. And although I read your posts more than once I do not understand why your approach should make things easier - on the contrary I am of the opinion that the things I have in mind (and I hope not only I have them in mind) would be even more complex to develop.

The data model for the "contribution process" needs to be developed entirely, we don't have nothing for this up to now. That's ok, the game data is more important and basic. But we have to define the process in detail and afterwards develop an appropriate data model for it - and then to application code on the server and the client.

Perhaps we do not only need a "horizontal prototype" but also a "vertical prototype"? (by "vertical" I mean from the bottom to the top and back in this diagram with authentification and a minimal contribution process). That could make things clearer for everyone and hellp us to detect problems or uncertainties.

[MZ per X]

I do not imagine that any information would be unavailable to any client. Rather, exactly one feature would be restricted to the official client: modifying the database.

I'm no developer, too, but I did a simple diagram to visualize the workflow which I think you suggest:

Oregami general workflow (Tracy Poff).png

(11.4 KiB) Downloaded 1538 times

Every client reads final data from the Oregami server, but only the official Oregami client can write final data after they have been approved there.

On the other hand, the users could use any capable client to modify the non-final, unapproved data within the Oregami client, obviously within the limits of our contribution and approval process there.

Did I understand that correctly?

I feel like I must have at some point written something horribly ambiguous in my first post, but I can't figure out what. It's quite bewildering and frustrating.

Please don't give up. I know that feeling from some data model discussions, complex proposals need some time to seep through. Not that I could be of further help with this one.

[Tracy Poff]

Since it seems that I have not explained myself clearly, please allow me to try again, giving a bit more detail to my proposed design. I apologize for the length of this post, but I want to be as clear as possible.

I'll break this up into descriptions of three parts of the Oregami 'ecosystem': the oregami-server; the official, central oregami-client; and third-party clients.

Responsibilities/services provided

oregami-server

• Interfaces with the game information database, which stores only information about games/companies/etc. That is, everything in the UML Model, but not what is in the Model for User Data.
• Provides a RESTful API that responds to HTTP GET/HEAD requests from any client (be it oregami-client, a web browser, or something else) without authentication, replying with data from the database.
• Provides a RESTful API that responds to HTTP POST/PUT/PATCH/DELETE requests from authenticated clients by modifying the database. I propose that only a single client will be acceptable, specifically the official instance of our client--only it will have the key.
• When a write request is made by an authenticated client, oregami-server makes that change in the database. Envers or something like it versions the database.
• A database field (or, I suppose, one per table) records who is responsible for the change, comments on the change, etc. This information is sent by the client making the change.

oregami-client (the central instance)

• Maintains a database of users, proposed changes, and potentially private messages between users or other user-related information.
• Authenticates users.
• Displays data retrieved from the oregami-server to all users, authenticated or not.
• Manages the approval process: this includes providing for commentary on changes, allowing users to pull unaccepted changes for further work, or whatever else may be a part of it.
• When a change has been approved, according to whatever guidelines and process we develop, sends the change to the server via oregami-server's authenticated-only API. Included in that data will be everything we want to save from the contribution process, including who is responsible for the change.

Third-party Clients

• Interacts with the oregami-server over the public, unauthenticated API.
• Displays and uses the data however is appropriate to the service the client provides.

Implementation details

oregami-server

The oregami-server should be relatively very simple, acting as a thin layer providing an interface to the database. Making the oregami-server as simple as possible will make it possible to be more confidence in its stability and correctness. We should expect that almost all traffic to the server will be interested only in reading data from the database, so it is most important that the server provide this function, and the less we have to touch the server due to changes in our contribution process, the better.

oregami-client (the central instance)

The client will consist of two parts. The first part is the backend: the database and the software that interacts with it, which stores information about users, proposed changes to the oregami-server's database, and whatever else falls in the client's domain, and manages the contribution process. The second part is the frontend: the user interface, which shall be implemented in HTML+JS, and which interacts with the backend via some API--potentially, a RESTful (or nearly-RESTful) API like the oregami-server provides.

Third-party Clients

Almost all clients will need only read-only access to the oregami-server. They will have it, without any authentication being required.

If some third-party client should require write-access to the database (which I doubt would be the case), it could communicate with the central oregami-client's backend API in the same way that the oregami-client's frontend does. The user of the third-party client would have to provide her credentials to the third-party client to facilitate this, unless something like OAuth were implemented by the oregami-client.

Benefits

• Except for schema changes to the database, it would almost never be necessary to touch the oregami-server, after it was fully implemented. This would dramatically reduce the likelihood of introducing bugs to the server.
• By having and committing to a simple and stable read-only interface on the oregami-server, third parties could trust that work put into interfacing with the oregami-server would not be wasted.
• Most development work would be focused on the oregami-client. Even if we somehow completely broke the oregami-client by some change, the oregami-server would continue to be available and provide the same services exactly as efficiently to all consumers of Oregami's data.
• It would be not only easy to make the server completely RESTful, it would actually be more complicated to do otherwise.
• Since the details of who is responsible for what contributions would be sent to the oregami-server by the oregami-client, these details need not be tied only to users of the oregami-client. In particular, if we imported data from some other database, the importer could send along an attribution to that database, or even the user of that database that was responsible for the data.

Well, that is quite a wall of text! I don't think that I can be any clearer about my idea than this, so I hope that this isn't too confusing. Of course, I'll be happy to answer any questions you may have.

[gene]

Wow - congratulations for the longest post so far (and perhaps forever)

Thanks for your explanations, I will definitely need some time to think about all the details and to prepare an answer.

During that time, it might be interesting for anybody who is involved here to watch the talk "breaking the monolith" by Stefan Tilkov. I think it is absolutely on topic! I saw Stefan with this talk about two weeks ago live here in Münster, Germany.

[Rola]

Not sure if this is related to User Data, but the future website should use just one login, linked with forum, just like at MobyGames.

I really dislike registering twice on the same site! (forum + site proper)

[gene]

So after I have read your (Tracy) last post multiple times, I must say that it helped me much to understand what you wanted to say
That does not necessarily mean that I totally agree with everything

And let me warn you: my post is not as well structured as yours. Sorry for that.

I will try to "reverse" your reasoning, and let's see if I understood your needs correctly. Afterwards we can discuss about the/a resulting technical architecture (or other things) with that we can fulfill your (or better: our) needs.

You mention that you want:

• to reduce the likelihood of introducing bugs to the server (with "server" you mean the mostly-read-only-database with game data)
• that third parties could trust that work put into interfacing with the oregami-server would not be wasted
• that if something on the "oregami-client" is broken then the oregami-server (game data) would continue to be available and provide the same services exactly as efficiently to all consumers of Oregami's data
• technical "imports" of data from sources which are not "oregami users" can be made with special attributions

So the three first things are important, of course. But do we really have to split our application in two applications to reach these goals? I cannot see why we should not be able to guarantee that one part of the application (read game data) is stable while another part of the application (contribution process, user management/features) is under development? And I don't think that we will mess up the machine/process so much that it will refuse to work completely.
Your last point (data imports from other databases) is only a "functional" thing. Of course it would be nice to offer a feature to import data from other databases with a special support, but again I don't see why this would lead us to the need to seperate our application in two.

If we split our application as you are describing it that would technically mean:
1) database server with REST interface (Java/REST + database)
2) user server with process logic and user persistence but without GUI (Java/REST with database)
3) HTML+JS application with GUI

That makes three applications to develop. I don't want to start three application servers (plus the two database servers) for development!?

Another reason why I don't like to split the "client and server" like you are suggesting:
parts of the datamodel must be doubled. Most of the game datamodel, as you can send in game data on your "client". And also parts of the user datamodel, as I want to store next to the game data which user entered the data, which user commented on which data, which user approved which game data and so on. You see: I don't want to seperate these data sets, because I thing they have too many relations.

Abour your OpenID-thing:
You say that OpenID is also a service that is seperated from "real" applications. That's true, but you neither can send private messages to other users through OpenID, nor can you do something more complex with it like entering complex data in a multi step process with several users involved. OpenId deals with authentification, but not with more. That works and we may also offer the usage of OpenID tp login to the oregami system.

[Tracy Poff]

So after I have read your (Tracy) last post multiple times, I must say that it helped me much to understand what you wanted to say
That does not necessarily mean that I totally agree with everything

I'm glad that we're making progress with understanding one another. And, of course, I don't expect you to agree with everything I say.

Let me preface this with a tiny mini-biography of me, to make my biases clear: I am not a professional software developer, though I do spent a fair amount of my time developing software for my own use. I studied math at uni, and I tend to enjoy functional programming (Haskell is a beautiful language!), so my problem-solving strategy typically involves breaking problems into pieces that can be solved by functions that are small enough to be 'obviously correct'.

Anyway: my immediate reaction to the idea of a single piece of software that manages a game database, provides a communication system (including maybe both private messages and forums), allows moderated editing, and who-knows-what else is that this is a piece of software that has too many hats to wear, and development and maintenance will be simplified by breaking it into multiple applications. So, for full disclosure, if you prefer developing big, monolithic pieces of software, we're going to have some philosophical disagreements.

Since terminology has been a problem, let me set out by clarifying my terminology:

If we split our application as you are describing it that would technically mean:
1) database server with REST interface (Java/REST + database)
2) user server with process logic and user persistence but without GUI (Java/REST with database)
3) HTML+JS application with GUI

Let me call 1) the database server, 2) the moderation server, and 3) the client. I recognize that the second may be handling things like messaging as well, but let's keep it as simple as possible.

I cannot see why we should not be able to guarantee that one part of the application (read game data) is stable while another part of the application (contribution process, user management/features) is under development? And I don't think that we will mess up the machine/process so much that it will refuse to work completely.

Well, of course with proper care and discipline, we should be able so sufficiently isolate changes that we don't make any huge mistakes and break unrelated parts of the server. But, every time we are touching the same codebase, we do have a risk (if slight) of breaking things unintentionally. This is especially the case if the different parts of the software are not really well separated. But, point taken: we don't have to have actually different pieces of software, as long as we're careful about keeping things loosely coupled.

Your last point (data imports from other databases) is only a "functional" thing. Of course it would be nice to offer a feature to import data from other databases with a special support, but again I don't see why this would lead us to the need to seperate our application in two.

This point was in response to the idea that you must have the user database on hand to do change attribution. If that's really true, then it means that you're doing it by linking against the user table, or something, and I was making a point that there were other attribution scenarios where that wouldn't be a good idea--and therefore you probably don't really need the user table on hand.

That makes three applications to develop. I don't want to start three application servers (plus the two database servers) for development!?

Well, it might be inconvenient for the moment, but if I dream that in the future we may have a very popular service, it seems to me that we might well want to be able to put the (relatively) high-traffic database server on a different machine (or in a different instance in the cloud, or whatever) from the (relatively) low-traffic moderation server. This especially if there's heavy development: we could bring the moderation server down for work as often as needed without disturbing the database server.

Another reason why I don't like to split the "client and server" like you are suggesting:
parts of the datamodel must be doubled. Most of the game datamodel, as you can send in game data on your "client". And also parts of the user datamodel, as I want to store next to the game data which user entered the data, which user commented on which data, which user approved which game data and so on. You see: I don't want to seperate these data sets, because I thing they have too many relations.

Here I disagree. The client that we have now does not need to duplicate the data model to display the data, and there's no reason why my proposed design would require it, either. I don't know exactly how you plan for the contribution process to look 'under the hood', so let me imagine an example of how it might work, to show why I think this way. (Note that the example isn't intended to represent our actual data model, etc. I just intend to show the process.)

Code: Select all

HTTP GET http://api.oregami.org/games/1

Response:

{
    "id": 1,
    "title": "Monkey Idland",
    "publisher":"Lucasfilm Games"
}

Oh no! A typo! To fix it, the moderation server might send back something like:

Code: Select all

HTTP PATCH http://api.oregami.org/games/1

{
    "title": "Monkey Island"
}

Of course, in my proposed design, the actual patch contents would have the change attribution and some other metadata, but you see my point.

Now, for the client to display the modified version before it's sent to the server, it just has to apply the patch itself--but at no point does it really need to know anything about the data model. It's just working with a JSON object.

Now, I agree that the client UI will need to in some sense duplicate the data model by providing forms to fill out to add games, and by knowing that a game has for example a 'title' attribute that can be displayed in the page title, but that's duplication that would occur regardless of where the data model is stored.

As for the other part, I don't think that the database server must store the user table, either. We could imagine that the 'attribution field' for the change is some free text field which can be populated by the moderation server on making a change with something like: "Modified by <a href="http://moderation.oregami.org/user/2733">Tracy Poff</a>. Approved by <a href="http://moderation.oregami.org/user/2309">gene</a>." Then for change history we have Envers or whatever providing the historical contents of the field... well, that's all just matter of implementation. The point is that we can certainly attribute things properly without the database server having the user table on hand.

Abour your OpenID-thing

I was just giving an example of authentication being handled by something other than the app that needs the authentication, so please forgive me if I don't pursue this any further. Of course I agree that the scenarios are different.

Now I've written another huge post, so big that I can't remember whether I said everything I wanted to when I started writing. I'm sorry about that--I don't mean to make it a habit.

It looks like, in the end, you think my idea will complicate development (and maybe have other drawbacks), and I think that it will simplify development (and maybe have other benefits). I'm not exactly sure where to go from here...