Difference between revisions of "Product Opener"

From Open Food Facts wiki
Jump to navigation Jump to search
(→‎Installation: installation on Debian)
(9 intermediate revisions by 2 users not shown)
Line 2: Line 2:
  
 
Product Opener is the software used by Open Food Facts to create and maintain its collaborative database of food products in open data.
 
Product Opener is the software used by Open Food Facts to create and maintain its collaborative database of food products in open data.
 +
 +
[[File:ProductOpenerStructure.png|thumbnail]]
  
 
== Open source ==
 
== Open source ==
Line 16: Line 18:
  
 
The data for each product is stored in a structured object (several levels of hashes and arrays). The data is stored on disk in individual files in Perl's binary format for objects (Storable). The history of each change to the product data is also saved in the same format. There is one binary file for each version.
 
The data for each product is stored in a structured object (several levels of hashes and arrays). The data is stored on disk in individual files in Perl's binary format for objects (Storable). The history of each change to the product data is also saved in the same format. There is one binary file for each version.
 +
[[File:ProductOpenerDataStructure.png|thumbnail]]
  
 
==== Product data index ====
 
==== Product data index ====
Line 21: Line 24:
 
The current version of each product is stored in a MongoDB database. The database includes indexes and is used to search and display products.
 
The current version of each product is stored in a MongoDB database. The database includes indexes and is used to search and display products.
 
MongoDB allow storing structured objects as-is, so there is an exact correspondence between Perl's internal format, MongoDB's format and the JSON export of the API.
 
MongoDB allow storing structured objects as-is, so there is an exact correspondence between Perl's internal format, MongoDB's format and the JSON export of the API.
 +
[[File:Perlmongodb.png|thumbnail]]
  
 
=== Product images ===
 
=== Product images ===
Line 46: Line 50:
  
 
=== Product Opener API ===
 
=== Product Opener API ===
 +
[[Project:API|See Detailed Page: Project:API]]
  
 
Product Opener provides a JSON API to search products and read their data.
 
Product Opener provides a JSON API to search products and read their data.
Line 51: Line 56:
 
There is also a limited API to upload product images and edit some of the product data that is used by the mobile applications.
 
There is also a limited API to upload product images and edit some of the product data that is used by the mobile applications.
  
The API functionality is currently provided by the Product Opener Web Server, but it may change in the future (e.g. at some point we could implement a full API for accessing (reading + writing) the Product Opener database, and have the Web Server call the API instead of going directly to the database).  
+
The API functionality is currently provided by the Product Opener Web Server, but it may change in the future (e.g. at some point we could implement a full API for accessing (reading + writing) the Product Opener database, and have the Web Server call the API instead of going directly to the database).
  
 
=== Product Opener client applications ===
 
=== Product Opener client applications ===
  
 
==== Apache Cordova mobile app for Android, iOS and Windows Phone ====
 
==== Apache Cordova mobile app for Android, iOS and Windows Phone ====
 +
 +
[[Mobile App Development Notes|See detailed article: Mobile App Development Notes]]
  
 
The current Open Food Facts mobile apps for Android, iOS and Windows Phone are developed with Apache Cordova (previously known as Phonegap) with the same HTML + Javascript (including jquery and jquerymobile) code base.
 
The current Open Food Facts mobile apps for Android, iOS and Windows Phone are developed with Apache Cordova (previously known as Phonegap) with the same HTML + Javascript (including jquery and jquerymobile) code base.
Line 67: Line 74:
 
== Installation ==
 
== Installation ==
  
(In progress.)
+
(in progress)
  
 
See [[Product Opener installation]]
 
See [[Product Opener installation]]
  
 
and [[Product Opener installation on Debian / Ubuntu]]
 
and [[Product Opener installation on Debian / Ubuntu]]
 +
 +
== Maintenance tasks ==
 +
* [[Product Opener - How to update a taxonomy]]
 +
* [[Product Opener - How to update interface translations in Lang.pm]]
 +
 +
== Documentation ==
 +
 +
(in progress)
 +
 +
[[Product Opener documentation]]
 +
[[Category:ProductOpener]]
 +
[[Category:Developer]]

Revision as of 20:02, 22 October 2015

Introduction

Product Opener is the software used by Open Food Facts to create and maintain its collaborative database of food products in open data.

ProductOpenerStructure.png

Open source

Product Opener will be released under an open source licence so that it can be reused by other projects, for instance to open the data for other types of products.

The Product Opener original code includes code related to the original developer's other projects. Efforts are under way to clean, decouple and document the Product Opener code so that it can be released as open source.

Components

Product Opener Database

Product data and history

The data for each product is stored in a structured object (several levels of hashes and arrays). The data is stored on disk in individual files in Perl's binary format for objects (Storable). The history of each change to the product data is also saved in the same format. There is one binary file for each version.

ProductOpenerDataStructure.png

Product data index

The current version of each product is stored in a MongoDB database. The database includes indexes and is used to search and display products. MongoDB allow storing structured objects as-is, so there is an exact correspondence between Perl's internal format, MongoDB's format and the JSON export of the API.

Perlmongodb.png

Product images

Images are saved on disk for each product. Original images uploaded through the web server or mobile applications are kept. Thumbnails are also generated for the final product image.

Aggregated data

Some of the data is periodically aggregated and saved in Perl binary files (e.g. average nutrition facts for products of each category).

Exported data

The data for all products is periodically exported in CSV and RDF formats.

Product Opener Web Server

The web interface for searching, displaying, adding and editing products. (e.g. http://world.openfoodfacts.org for Open Food Facts)

The server is written in Perl. In production we use a light weight Apache web server for static files (e.g. images) that does reverse proxying to a modperl Apache server that dynamically generates the HTML pages.

The form to add and edit product uses Javascript and jquery.

In the backend, other libraries and software are used, such as Tesseract for optical character recognition for ingredients.

Product Opener API

See Detailed Page: Project:API

Product Opener provides a JSON API to search products and read their data.

There is also a limited API to upload product images and edit some of the product data that is used by the mobile applications.

The API functionality is currently provided by the Product Opener Web Server, but it may change in the future (e.g. at some point we could implement a full API for accessing (reading + writing) the Product Opener database, and have the Web Server call the API instead of going directly to the database).

Product Opener client applications

Apache Cordova mobile app for Android, iOS and Windows Phone

See detailed article: Mobile App Development Notes

The current Open Food Facts mobile apps for Android, iOS and Windows Phone are developed with Apache Cordova (previously known as Phonegap) with the same HTML + Javascript (including jquery and jquerymobile) code base.

The Android app uses Moodstocks image recognizer and barcode scanner plugin. The iOS and Windows Phone apps use the BarcodeScanner plugin.

The code for the Open Food Facts phonegap app is already in open source:

Installation

(in progress)

See Product Opener installation

and Product Opener installation on Debian / Ubuntu

Maintenance tasks

Documentation

(in progress)

Product Opener documentation