Software Analysis and Design Wiki

Now that the first funding is secured from the Swiss Prototype Fund, it’s a good time to start a more detailed software analysis … so everyone knows what to develop. For that, I’ll take a step back and re-visit some of the assumptions, and also evaluate how to fit the software requirements with the development budget we have from Prototype Fund.

Table of Contents

1. Approach to architecture
2. Planned software architecture
3. Other architecture options

1. Approach to architecture

Software development is always quite some effort (and expensive!), so generally a piece of software should re-use as much existing work as reasonably possible, and should itself be re-usable by others and for other purposes. This is especially true for this software, as it is publicly funded and open source, so already geared towards re-use.

This brings us to a pragmatic approach to software architecture that has only two rules:

  1. "Find the most complete open source base software and use it." That’s the starting point for the software architecture decision here. In the following sections, we explore the different alternatives on which this software could be based.

  2. "If you have to write code, make it solve not one problem but a whole class of problems." Could also be expressed as a variant of the Golden Rule for writing open source software: Try not to write code. But when you have to write code, write software that you would like to find when looking for a suitable open source base software.

For the second rule from above, we need a concise high-level description of what class of problems we are trying to solve with our software. “School electrification projects in Uganda” is not a valid class of problems but rather a specific one. Ideas for how to describe our class of problems so far:

  • Microfinance software. Microfinance shares many of the same challenges as what this software is trying to solve: there are one or several funders and a multitude of recipients; there is a requirement for scrutiny and proper use of finances provided, whether they are loans or grants; and there is a disconnect between the bureaucratic, formal world of funders and the lived reality of recipients, where our planned innovations like monitoring and validation via videos can help.

  • Crowdaction software. This is basically the reverse of crowdfunding software: instead of multiple funders and one central action or project, there is one central funder and multiple actions or projects, executed by the crowd. It can be seen as another type of crowdsourcing.

  • :bulb: Freelancer marketplace for non-digital remote services. Currently, a freelancer marketplace like Upwork or Fiverr is used only for projects of which the results can be delivered digitally. Other service marketplaces are dedicated to non-digital projects, but then always for local projects. For example, contracting handicrafts work in the case of MyHammer in Germany. A marketplace dedicated to remote and non-digital services does not seem to exist yet. It has a unique challenge related to the verification and quality assurance of the work, here to be solved with video messaging.

    In the ClimateGains project, the marketplace feature that allows to have multiple clients for projects / funders in parallel would not necessarily be used. But it would also not hurt to have it and could be useful for potential re-uses of the software. Such alternative uses may include, among others: a marketplace for remote freelance services around property management, garden maintenance, agriculture / silvoculture, construction, beach cleanups and similar environment protection activities.

2. Planned software architecture

As discussed above, software architecture starts pragmatically from the most suitable base software, and everything else “falls into place” once that decision is done. We estimate that the UNDP Online Platform for Voluntary Bilateral Cooperation is the most suitable base software, for at least the following reasons:

  1. We can easily reuse their open source carbon coop software as the backend in our Prototype Fund project. It has a generic workflow builder inside, which already prepares our software to also cover other use cases beyond Ugandan schools in the future. It also probably has all the reporting features that funders want.

  2. We contribute our changes and extensions of the carbon coop software back to UNDP. Could lead to interesting opportunities related to the Article 6 activities in the future. Relevant extensions would include the verification of submitted images, videos and GPS coordinates, and reports that include links to these elements.

  3. We (mostly @owen as I understand) build a custom mobile app that acts as the “frontend” for field users, and talks to the carbon coop software using its existing API. This custom mobile app would use web technologies plus Tauri (or similar) packaging into a mobile app, but would also include other code that accesses native features of the mobile platform and that enables an offline-first functionality. Means, the app would be happy with being offline, and will sync collected data only once it finds a reliable online connection.

  4. We prepare our custom mobile app to be easy to generalize (though potentially at first only applicable to the “Ugandan schools” project), based on the forms that can be designed in the carbon coop software. However, these forms would be translated (semi-)automatically into prompts for video responses, and the app only exposes the video-centric interface that we want it to have. This makes the app into a tool that lets activists interface with old-school funders.

  5. :bulb::bulb: We can now lobby governments that are buying emission offsets on the UNDP carbon coop platform to update their processes by using our custom mobile app for their project. Adapting and extending our mobile app as needed for these clients will help improve that app, and would provide a means of funding for us developing it further.

    I’m not proposing to drop the concept of a ClimateGains platform. This lobbying can be in addition to running our own platform, using a copy of the carbon coop software. Either that lobbying or our own ClimateGains platform or both may be successful. But only with an architecture plugging into the UNDP carbon coop software we can do that lobbying.

  6. Being compatible with the UNDP carbon coop platform means also being compatible with host-country government platforms. This also solves one major problem that we can’t address ourselves, namely double counting-claiming: somebody pitching the same project verification video to two different donors through two different channels. This is solved when the host country government has a natural monopoly on keeping track of all climate actions, and the easiest implementation of that is when all software solutions integrate with the UNDP carbon coop platform. While this may not be relevant for the Uganda school projects in the foreseeable future, it will be relevant for the majority of projects and all governmental funded ones.

From our first call with the developers of the carbon coop platform, here is a short description of their own software architecture. The official name is “UNDP Online Platform for Voluntary Bilateral Cooperation under Article 6 of the Paris Agreement” and it consists of two parts as follows. Only the first part, the “main backend software”, is the one we would re-use:

  • main backend software (this is the software called “Platform for Voluntary Bilateral Cooperation” and accessible under carboncooperation.undp.org; in the following called carbon coop software by me)

    • This main backend app is not meant for a smartphone. Desktop-only layout.
    • Open source, code is published already.
    • AngularJS frontend, NodeJS backend, MongoDB database, Azure server infrastructure. Frontend and backend talk to each other by API.
    • Uses MongoDB because forms are always different in the different processes that can be configured in that app, so data structures are also always different.
    • Has a full workflow customization process that allows to set up multiple processes (“projects” in our case), put steps into a required order, repeat steps, and have steps that are implemented by external, custom applications. The first use case for the latter is the rice farm monitoring web app they develop now. The connection of these custom apps to the main backend app is not yet decided, but will probably re-use the API between Angular.JS frontend and NodeJS backend of that main backend app.
  • monitoring app

    • For the monitoring specifically for the rice farmer project, a separate responsive web app is now in development by the same group of developers.
    • It is not realized as a configurable workflow inside the main backend app because (1) the steps it covers do not have to change over time, so that flexibility is not needed but on the other hand (2) unlike the main backend app, this app needs to be usable on by farmers, including on smartphones and not just with a desktop browser.
    • The new monitoring web app must include photo and video evidence. Embedded GPS coordinates will count as evidence.
    • The new monitoring web app will be rather similar to a spreadsheet, as that is what farmers have been using so far for monitoring.
    • There will be a backend inside this monitoring app for people to check that GPS coordinates match to farmer addresses etc…
    • All output will be open source. Will be published as public good on UNDP roaster.
    • Already published in a Github repo.

With their “main backend app” selected as our base software, and after some smaller architecture decisions downstream, our software architecture now looks like this:

Some explanations and justifications:

  • The software architecture block diagram shown above shows software components as blocks, forming layers one above the other. Any higher layer block depends on functionality provided by the lower layer block (or blocks) on which it rests.

  • Unlike in earlier proposals, there is only one piece of software for field users, called “Frontend UI”. It appears to be a native mobile app and can be installed from the Android app store as any other Android app. However, technically it is a glorious website, packaged as an app running locally on the phone. It can access native Android APIs for camera use, video re-encoding and the like via the Ionic or Tauri frameworks used to package a web app into a mobile app.

    The difference between this technology and a true native app does not matter to a user, but is a tradeoff from a tech point of view: it leads to a more resource hungry application, but on the plus side it can be adapted faster and more easily, it is easier to find developers for doing changes, and unlike with true native mobile apps, it is easily possible to implement any design requirements.

  • Our frontend UI is meant to be a generic tool that can translate any project management process form defined in the backend UI to a fully video-based process for the field users. Probably our first versions will not be that generic and limited to the Ugandan school projects, but the software design should be prepared already to allow extending it towards this generic use case later. This is different from the approach that the UNDP carbon coop platform devs are now taking with the rice project monitoring app, where they develop a custom app for just one monitoring step in one specific project. In a way, their approach is the prototype of monitoring with visual evidence, and ours would try to be the production version, easily usable for all projects that the carbon coop platform may manage later.

  • The “Backend UI” component is an extended version of the user interface of the UNDP Online Platform for Voluntary Bilateral Cooperation, and kept compatible with it by contributing changes upstream.

  • The “server” component is an extended version of the user interface of the UNDP Online Platform for Voluntary Bilateral Cooperation, and kept compatible with it by contributing changes upstream. Its choice of database is due to what the current version uses – which is a tradeoff, as usually schema-free databases like MongoDB are better avoided.

  • Different JavaScript UI frameworks are used for the frontend UI and backend UI components, namely Vue.js and AngularJS. This due to what the different groups of developers are used to, and due to Vue.js being our internal standard for all JS software development in Edgeryders. At least both are JS frameworks, so the same developers can work on both parts after some days or weeks of getting used to the other UI framework. And perhaps other groups of developers might contribute other third-party components to the carbon coop platform later, which would further normalize this decision as each group would again choose their favourite UI framework over AngularJS.

3. Other architecture options

Before settling on the architecture as described above, we also had considered various alternative starting points and architectures based on them, as follows:

  • PayCoupons marketplace software. Modified to be a freelance marketplace software plus video messaging. This is quite an interesting option but I’d have to talk to @daniel if we can make this possible. In short, PayCoupons is a service marketplace software that Daniel and me developed, using a circular barter algorithm to make exchanges possible without money. The PayCoupons platform itself is in hibernation right now as we were not able to get enough traction for a non-monetary marketplace platform. While the barter system is not relevant here and would be dropped, the interesting thing here is that our software includes all major marketplace features already incl. requests for work, pre-packaged services that a service provider can choose to offer, sub-communities, community management functions and so on. And, we can vouch for the technical quality of this software and Daniel, who did most of the programming, knows his way with this software and is fast in adapting it; that is helped by the fact that PayCoupons is made with Ruby on Rails, which is geared towards rapid application development, and that Daniel has more than ten years of experience with that. The open question is how and under what conditions we can make this software open source.

  • Freelance marketplace. Based on an existing, open source freelance marketplace software, if such a thing exists.

  • Crowdfunding software. Based on an existing, open source crowdfunding software; while a freelance marketplace software might not exist open source, there are definitely two open source software products for crowdfunding platforms: the one developed and used by Goteo, and a Brazilian one that I’d have to look up again. For ClimateGains, climate actions would be presented like crowdfunding projects, and instead of the “contribute” step people would get a “I want to do this” process, which would be an additional feature to add.

  • Business workflow software. Based on an open source business workflow software (e.g. based on Odoo or another well-known open source ERP system)

  • Forum software. Based on an open source forum software (basically Discourse, with the added advantage that our custom software Open Ethnographer is already integrated with Discourse)

    In a nutshell, Discourse is only suitable when you keep the Discourse UI and just add little things to it. That’s because the Discourse UI is hard to adapt but comparatively easy to extend. In other words, if the user actions follow a different paradigm than “it’s a forum discussion”, then a custom application or other base software is more suitable.

    The server side component could also be based on Discourse, but could also be a custom application for workflow management and reporting. If using Discourse, then the server-side component would be a major component that runs alongside Discourse, re-using its database (but having its own tables) and re-using its login flow. Similar to how Open Ethnographer is implemented right now. Discourse could be suitable, but only if enough additional arguments are in favor (such as, re-using Open Ethnographer, integrating with the team communication tool etc.).

  • Messenger software. Re-using an existing messaging app that can already handle video. For example, Signal is open source and has a “push and hold” feature to send video messages. The backend UI (“admin interface”) would then be based on a web version of the same messaging software that the frontend would use or build on. Sadly, Signal does not have a web version, only mobile and desktop versions.

    It seems possible to base the frontend software on the Signal open source messenger application. This will provide a familiar and complete chat-style interface. Especially, Signal has the “hold for video” button for video messaging.

    The challenge with this is how to provide the project selection options that the activist side app also should provide. Since Signal is not very well-known however, people will not already have it installed. And when we have to tell them to install a mobile app anyway, it could also be an adapted version of Signal that includes additional UI for the project selection, project status overview etc…

  • Camera software. In this case, the architecture would start from selecting a component for the part of the software used by people in the field. The current UI design for both the activist and validator interface is a video-centric application and does not include any part of (say) a typical forum user interface. So it is possible to implement it as a custom mobile application, based on a video recording app like OpenCamera, providing both the activist and validator features in the same software. It will talk to a server-side component and may indeed use Nextcloud for syncing the video files. Since only simple video recording is required, OpenCamera might also not be the best base application: simple video recording is easily available from various libraries and widget sets, avoiding the lock-in to the rather ancient software architecture of OpenCamera.

  • Fully custom web-based software. Built with RAD (rapid application development) technology such as Ruby on Rails, providing both a backend UI and a server-side API to use by the frontend (if needed). For the server-side component, any RAD (rapid application development) technology is suitable, as it has to be easily modifiable but not overly pretty. For example, we have extensive experience with Ruby on Rails. It’s possible to create Ruby on Rails software that functions both as a piece of Discourse but can also run stand-alone; Open Ethnographer is already close to that.

1 Like

Everyone @Coordinators, please provide your feedback and ideas in this discussion of the ClimateGains software architecture. I’ll edit the results into the wiki post at the start of this topic.

Right now, I’d be interested to learn especially:

  1. If you think I missed any basic alternative for a possible software architecture of the ClimateGains software?

  2. How important it is to include the Open Ethnographer software? This relates to the software product as it is in its current technical form, not just to its features, which could be implemented in an adapted form into any other software product. It’s rather about the “politics” behind including Open Ethnographer: is it essential for future fundraising or other reasons to build on this Edgeryders software product? This question is esp. for @nadia.

  3. Similarly, how important is it to include NextCloud in the software architecture? Again, this is about the politics behind using NextCloud, not about its features, which may be provided also by other pieces of software.

1 Like

Just to be perfectly clear: I suggested the technologies based on “stuff I use”, not a deep analysis of suitability. Feel free to deviate from the proposal / improve if!

  1. I’ve been flirting with PeerTube as an alternative to Nextcloud for the video backend, but don’t understand enough about the technology to judge viability and worth of it. AFAIK PeerTube uses Bittorrent for video distribution and thus should allow for slimmer servers than direct downloads via a Nextcloud (only relevant at much larger scale)?

=> This should also answer 3): Nextcloud is only in there because I use it privately, other options are welcome. Having said that, I think they got fairly large funding from public sector entities, making me believe they will continue to exist and improve long term.

My opinion on two: I don’t think this needs to be part of the prototype necessarily.

1 Like

Thanks for the freedom to think about and re-think the tech aspects. I think it’s worth it … already just had one interesting new idea how it could all be done both fast and well (Ctrl + F “PayCoupons” in the wiki above). Just ideas for now, of course.

That’s interesting, didn’t know it is based on BitTorrent. Will have a detailed look at PeerTube. Preliminary opinion though: server load is also not huge when all videos would be hosted right on the server, such as with NextCloud, and served from there. Because each of the videos will only get a few views (10-30?), not millions. The limiting resource is rather storage on the server, but that is relatively cheap and easy to extend, and even storage is not needed in crazy amounts when choosing resolution and compression adequately for the purpose of project documentation.

1 Like

Just had the most productive online meeting in at least a year, with @OmaMorkie and the developers of the UNDP Online Platform for Voluntary Bilateral Cooperation under Article 6 of the Paris Agreement.

I have added my meeting notes about their carbon coop platform software to the wiki above, and also a new proposal there to choose that piece of software as the basis for ClimateGains. That’s my favourite architecture option right now!

We will also get a post-meeting e-mail with links to the repositories etc…

1 Like

RE: :bulb::bulb: Being compatible with UNDP means being compatible with host-country government platforms. This also solves one major problem (double counting-claiming) that we can’t address ourselves (i.e. somebody pitching the same school video to two different donors through two different channels => Host country gov has a natural monopoly on keeping track of all climate actions).

While this may not be relevant for Vanessa in the foreseeable future, it will be relevant for the majority of projects & all governmental funded ones.

1 Like

Good one, added it to the wiki at the start of this topic now. I also added more documentation there about my proposal for a software architecture, including a little illustration and some notes on the tradeoffs implied in this achitecture.

@owen also agreed on that architecture in our call today. If I don’t hear anything to the contrary from you or @nadia, we’ll go forward with this architecture.

2 Likes

tempted to just stay silent in appoval, but as far as I can see that is all very reasonable. So from my that’s a big:

Gogogogo gl & hf