Contributors mailing list archives

contributors@odoo-community.org

Re: Recommended Contributor Tooling (RCT)

by
dar@devco.co
- 19/10/2015 15:25:33
Interesting funfact about python pip usability on windows: https://gist.github.com/blaggacao/b5e8fbff4eb16cbd1457

El dom., 18 oct. 2015 a las 21:53, Raphaël Valyi (<rvalyi@akretion.com>) escribió:
David,

when you talk about advanced Docker cloud deployment, I think you will also come to the conclusion that development and deployment have different constraints and for that reason you will use different base images. and launchers.

We start our stack on Heroku base image both for Voodoo and "Odoocker" which is our prod Docker base image. So they do have the same ancestry and we apply some of the same pre-packaging scripts (in akretion/voodoo) to provision both base images. But I think the needs are different so the final image and launcher should be. For instance we bundle the Postgres server in the Voodoo image because that's much smpler for dev. That's is nice for dev, but absolutely not for production... In short we transform Postgres in Sqlite in Voodoo...

Also many things in Docker assume stable mature software and light install. Odoo is extremely heavyweight with its 400 Mo shallow repo and it's also extremely buggy and moving target. so much that anything not considering git for deployment will not lead to viable projects.. Odoo evolves every year or so and need too because it's suboptimal code (but hey nobody did a better open source ERP, that's the point) and then all the localizations and OCA addons are evolving too trying to follow the evolution of the new release, so it's perpetually evolving and never stable. So many so called Docker best-practise that may allow you to deploy and scale say Redis or RabbitMQ will not apply to Odoo (not even telling every project is different because every company is different while out of the box Odoo isn't very appealing).

Regards.

On Mon, Oct 19, 2015 at 12:22 AM, David Arnold <dar@devco.co> wrote:
I think we definitly should schedule a hangout the coming week, to see how we can ge this aligned..

I have a question too, what is conceptually speaking the role of devstep in this? I fell I'm still ignorant on this point... :/

I played with Devstep and installed its Golang launcher. I fail to see how it's more portable than doing pip install voodoo. User has to download Python first? Well, if the guy want to look at Odoo we can assume he is able to download Python on windows or assume it's a troll otherwise or assume he would just simply click on some cloud demo otherwise. As for Linux/Mac it's by default... Is downloading python such an obstacle that it's worth the effort of rewriting and maintaing the Voodoo launcher in Golang? I doubt...
I'd like to answer with an interesting fact: docker is written in go. But in order to get started in with docker development, you never have to install golan on your machine. I think the main benefit, is that it lowers the entry barrier, true also to troll, noobs, etc. But, true, also to programmers which have worked with other languages so far. In the end, there is a point about resetting my machine every 2 month, or being able to start hacking on a windows surface pro during the next 13 hours flight, without major bother...

Now Vagrant is just a single cllck to install on Windows, Mac and Linux. How installing Docker on Windows and installing a Voodoo Golang launcher is simpler than just doing "vagrant up"?
This has a very precise answer:
Common to docker and vagrant - both need an installation of an hypervisor, which is a real big PITA, but we can't help that point, just hopes things get better.
difference to docker and vagrant: vagrant is installd on the host machine, docker isn't. But, wait. The actual question should be:
difference between docker-machine and vagrant?
Assuming, that both have binaries, there is no real difference in the user experience, despite of having to go to yet another website. But, internally docker-machine is much more tightly integrated with all that advanced kubernetes, cloud deplyment, rancher and flocker stuff, we (more advanced docker user) eventuall want to capitalize on. So this is where my tendency towards docker-machine comes obvious...

Now what we could absolutely do too is have a degraded Python Voodoo run command that launch Docker directly without the Docker compose benefits when Docker compose is not available, a bit like I suggest with the Vagrantfile. Then it's between installing Vagrant or the Voodoo python package...
That would be just perfect, but I wan't to verify fist, If I couldn't come up with a even better idea. the privileged dockers in a priviledged docker, this way we can shift off this dependency to the docker docker as well... :) However, I need to think this through, first..

El dom., 18 oct. 2015 a las 21:08, Raphaël Valyi (<rvalyi@akretion.com>) escribió:
But one doubt however:

I played with Devstep and installed its Golang launcher. I fail to see how it's more portable than doing pip install voodoo. User has to download Python first? Well, if the guy want to look at Odoo we can assume he is able to download Python on windows or assume it's a troll otherwise or assume he would just simply click on some cloud demo otherwise. As for Linux/Mac it's by default... Is downloading python such an obstacle that it's worth the effort of rewriting and mainaintaing the Voodoo launcher in Golang? I doubt...

Now Vagrant is just a single cllck to install on Windows, Mac and Linux. How installing Docker on Windows and installing a Voodoo Golang launcher is simpler than just doing "vagrant up"?

Now what we could absolutely do too is have a degraded Python Voodoo run command that launch Docker directly without the Docker compose benefits when Docker compose is not available, a bit like I suggest with the Vagrantfile. Then it's between installing Vagrant or the Voodoo python package...

On Sun, Oct 18, 2015 at 11:50 PM, Raphaël Valyi <rvalyi@akretion.com> wrote:
Hello David,

some answers:

  1. I would be open to having voodoo installed inside the Docker image and launch it via "inception". Now, I'mo not sure this is easy. One thing is that volumes shared with Docker (via -v) should be absolute paths, not that easy to achieve in a portable way on both Windows and Linux (you should share you project workspace and the shared odoo local repo at least).
  2. I'm totally open if you want to rewrite Voodoo in golang or hack into the Devstep launcher enough to get it able to Launch the Voodoo image with the proper options. It's just that everubody uses Ubuntu at Akretion and we know Python so it was just about getting fast to the point.
  3. In any case Docker compose is absolutely useful (but we could accept not having it in a degraded windows mode). It's really a design shame that Odoo is such a monolithic design, specially now that CMS crap. At Akretion we do several project with a Ruby web part or even Apache Solr and sharing them with Voodoo is just so much easier. Docker opens up a whole new world of micro-services. Launching a configured Eclipse+PyDEV or a Database browser are quite useful for the noobs.
  4. We deploy to the prod via git push ("git deploy"), we have special git hooks on the other side of the network. Now Odoo hosting in production is a whole subject in itself with many possiblities (shared host, isolated containers, auto-scaling cloud etc...) and is out of the scope of Voodoo which aims to be just a development tool. I suggest you contact me privately if you wish to collaborate seriously on these other topics.
Regards.
On Sun, Oct 18, 2015 at 11:23 PM, David Arnold <dar@devco.co> wrote:
Raphael, thanks a lot for your detailed explication!

This gives me the possibility to ask some more detailed questions, I'm interested in:

Voodoo is basically a pre-built dev environment bundled a single Docker image. We think it's the right mileage between the pre-build-dependencies for a turn key startup and the flexibility of Anybox recipe to deal with any module branch and Odoo branch, just like using the recipe without Docker. 
I wanted to open up discussion on this point because of the following reason: travisfile2dockerfile by vauxoo permits using the travisfiles, that since the implementation of travis by oca are present (!) and maintained (!) in every repo. travisfile2dockerfile adds therefor a dynamic environmentcreation, which is the same goal what you want to achieve with anybox, if I understadn you right. On the other hand, and this is the elegance I was aiming for, Moses probably can easily rewrite his t2d library to work as a maintenance script, that automatically transforms and updates (via PR and Github Bot Account) each repo's individual "last mile dockerfile" (based on a common oca base image). This way, I can easily implement and transform the docker contribution guide and adapt it to the oca ecosystem. The only host dependency is indeed git and docker, this way. (and docker-machine abstraction handler on OSX and Windows).

It makes projects very lightweight, self-contained (even the dev database is inside the project folder, just like Rails with sqlite) and extremely easy to share between developers. And we also have easy tools to push such a dev env to the cloud in just one command.
I assume this command is the generic docker export / docker import or some lightweight scripting and scp streaming arround it? Or can be acomplished that way? As I understand you, you simply persist the database volume onto the host for direct acces? This might be a good and desirable idea, and is supposedly accomplished by docker -v command, right? I take note of that, to include it into the contribution guideline, I'm aiming at. 

The python package of Voodoo is just a handy launcher. initially it was 20 lines of bash but then my co-workers rewrote it as a Python wrapper around Docker Compose, giving it unlimited flexibility to attach custom micro-services such as a mail catcher, a pre-confgured Eclipse, a database browser, a Solr server, a Rails e-commerce, anything you want, specified in the same project repo.
That is a very noble intention. I would however point out, that it is important to not depend on any local host dependency. But that wouldn't probably be a problem, you can easily rewrite that and include it into the docker host itself. This is called docker inception technique. It would feel something like docker run --privileged aktretion/voodoo commandxyz. Neither bash nor python are good enough for the local host, because of portability. But I feel we do not need any tooling, one git and one docker command is completely enough (just 2 standard BINARY commands) To setup the perfect dev environment. Can those microservices not be achieved by a standard docker-compose definition? We could move the "necesary" to a docker in docker "privileged" architecture, such as db, debugging and the odoo environment and leave optional microservices to an still optional docker-compose (like this windows user are not doomed).

Now yes, this depending on the maturity level of Docker compose on Windows and Mac, the launcher may not work on these platforms. Well in that case, in a few hours of work we can ressucitate the Vagrantfile launcher that will mimic the 80% of important features to fire the Docker workspace container on these plateforms.
I'm actually against this approach, because it add's even more unecessary plumbing and tooling to the local host. I'm very against this, because it increases setup times. Actually I have in my mind a design, where we would not need docker-compose, so this would be elegantly resolved by "reduce to the max" strategy.

I believe this is what you wanted to achieve. And yes it's mature, we have been more than a dozen of people using it for 6 moths now at Akretion...
That is definitely a valuable argument, however, if we could reach out to completely cut out this layer at all, which needs stabilization, why shouldn't we?

Best Regards, David

El dom., 18 oct. 2015 a las 19:38, Raphaël Valyi (<rvalyi@akretion.com>) escribió:
David,

Voodoo is basically a pre-built dev environment bundled a single Docker image. We think it's the right mileage between the pre-build-dependencies for a turn key startup and the flexibility of Anybox recipe to deal with any module branch and Odoo branch, just like using the recipe without Docker. It makes projects very lightweight, self-contained (even the dev database is inside the project folder, just like Rails with sqlite) and extremely easy to share between developers. And we also have easy tools to push such a dev env to the cloud in just one command.

The python package of Voodoo is just a handy launcher. initially it was 20 lines of bash but then my co-workers rewrote it as a Python wrapper around Docker Compose, giving it unlimited flexibility to attach custom micro-services such as a mail catcher, a pre-confgured Eclipse, a database browser, a Solr server, a Rails e-commerce, anything you want, specified in the same project repo.

Now yes, this depending on the maturity level of Docker compose on Windows and Mac, the launcher may not work on these platforms. Well in that case, in a few hours of work we can ressucitate the Vagrantfile launcher that will mimic the 80% of important features to fire the Docker workspace container on these plateforms.

Imagine that Voodoo is just Devstep for Odoo:

I believe this is what you wanted to achieve. And yes it's mature, we have been more than a dozen of people using it for 6 moths now at Akretion...

Cheers.

On Sun, Oct 18, 2015 at 5:38 PM, David Arnold <dar@devco.co> wrote:
Example, according to documentation on an XYZ python library:

$ python setup.py develop
Traceback (most recent call last):
  File "setup.py", line 11, in <module>
    from setuptools import find_packages
ImportError: No module named setuptools

Sorry, you just don't want to burden this onto someone who just want's to get started. Even to us it might be an obvious error. It's not the user's fault, that this is not working. It is the programmer's (as almost always) ...
El dom., 18 oct. 2015 a las 14:24, David Arnold (<dar@devco.co>) escribió:
Ps.: If you can ship me a windows 64 compiled binary, I'm happy to evaluate it thoroughly!

El dom., 18 oct. 2015 a las 14:22, David Arnold (<dar@devco.co>) escribió:
Hi Raphael,

thanks for your feedback. Indeed it is possible. Actually I'm still waiting more detailed technical description of voodoo to be able to draw my own judgment. A statement  "will be exactly what you want" assumes that you "understood exactly what I want". What I saw from a quick look it is a python library. So unfortunately this is not a common enough ground for tooling, mainly because it leaves higher level tooling (like python and virtualenv) to the user. This is not nice and not even necessary. Please correct my, though, if I understood the functioning of voodoo wrong. Again, I would be delighted by a short technical description comprising at least a description of what issues are addressed by what strategy. That would be really helpful.

Thanks and best regards, David

El dom., 18 oct. 2015 a las 13:08, Raphaël Valyi (<rvalyi@akretion.com>) escribió:
David,

sorry but I really think if you help contributing a Vagrantfile launcher for Voodoo it will be exactly what you want and following best field practices...

Without help we will make that Vagrantfile (unless Docker compose run smooth on Windows) but help could speed this up.

Regards.

On Sun, Oct 18, 2015 at 5:23 AM, David Arnold <dar@devco.co> wrote:
Moses, we have the problem, that most windows user, that install git via he official installer will opt for the (recommended) option that files are checked out with CRLF. I have the gut feeling, that if they have done so, bindmounting those files will produce non understandable errors in the outside layers of their stack for them. This would be another argument for having the git proxy and working over a network on the docker filesystem. Or we populate respective sources into bindmounted directories AFTER the startup in the run script... I mean to be able to leverage docker volume and OS standard tooling instead of having to especially prepare the IDE. And keep that stuff out of the odoo environment.

El dom., 18 oct. 2015 a las 1:47, David Arnold (<dar@devco.co>) escribió:
Hi Eric, thank you for your feedback!

What I want to achieve is a standard way for new contributors to get coding quickly, weather they be on windows, osx or linux. My role model newcomer does not know python, nor git, nor does he know other stuff. He knows barely some basic concepts of the command line, and still he is valuable, because he might be able to copy paste some code and get his business case working. And, he might grow and evolve...

Installing python is cumbersome and litters your computer, and after you do a computer reset (I do that every 2 months probably) you have to start again setting everything up. Setup virtualenv is even more cumbersome, actually I never got a hold on it. So be python based solution discarted. We need binaries to get people going. Enter docker, it ships binaries. But I'm not getting at deploying. I'm aiming at the perfect, most cross platform, easy, dead simple, get going in less than 5 seconds development stack.

The point in the above mail is derived from Moses's (i feel) briliant idea to reuse the maintainance work, that is already present in the travis files. He wrote a tool, that transforms it into a buildable docker file (https://github.com/vauxoo/travis2docker). This set's up a complete environment (python deps, odoo deps other deps). So if you want to start hacking on some oca repo, you just have to clone it and build the automatically maintained dockerfile which includes the specific environment of this specific repo. And maintainers would carry on maintaining the travis files only, which have kind of the historic precedence rights on their side.

After building and docker run, you can hapily modify the src on your host and everthing get's bindmounted or synced into the docker, so you go to localhost:8069 and see what you have done instantly (with pynitify and odoo autoreload).

That's the kind of instantaneousness I want to see, also on windows systems. And I have not mingled one single time with anything of the kind of virtualenv, pythondeps, git blabla, missed module dependencies. It's just something like git clone && docker build . && docker run. Required binaries: git, docker. That's all - available for nearly any thinkable of common platform. And you are good to go hacking on your working directory.

Not: This is quite agnostic of whatever (probably git based) upper continuous integration workflow you chose. But this is anyhow not a concern of the one how just wants to get his hands dirty.

I'm trying to collaboratively discuss the best design for such a tool and am committed to come up with a beta-worthy solution for testing. Maybe I need some help here and there, but with communities support, I think it's doable.

Best, David

El sáb., 17 oct. 2015 a las 22:08, Eric Caudal (<eric.caudal@elico-corp.com>) escribió:

Hi David,
We appreciate your  various contribution but in this case, I am not sure to understand where you are getting at.
Docker is very nice environment and I am in favor for the deployment although the techno is style young is not completely bullet proof.
We have our own here based on OCB and refreshed  every week (as of next week) : https://hub.docker.com/u/elicocorp/.
You can grab docker-files here:https://github.com/Elico-Corp/odoo-docker
(I will send a separate mail anyway to check if community could be interested in maintaining this repo in OCA)

But I dont see the point of your exercise. Current infrastructure is solid for travis: once your code is solid and your docker file is well structured, there is no need to travis the dockers (IMHO).

Resources are limited and we do not want to tumble our infrastructure that we have needed month to stabilize so we need clear objective and pros for the community and you will come with a fully prepared solution.
We barely have resources for maintainance and cannot really just TRY things...
If you can come with solid project definition and proof of concept + some resources to maintain it in the community, we could consider it

--
Eric Caudal [Founder and CEO]
Skype: elico.corp. Phone: + 86 186 2136 1670 (Cell), + 86 21 6211 8017/27/37 (Office)
Elico Shanghai (Shenzhen/Singapore) Odoo Gold Partner, best Odoo Partner 2014 for APAC
On 10/18/2015 10:37 AM, David Arnold wrote:
<blockquote cite="mid:CAOLEt-EMOGXd9JjKassUDD6zkzOD5ZFfsgcop13qFMD1TpJM9Q@mail.gmail.com" type="cite">
Having talked with Moses (vauxoo) bilateral about his solution further, I would like to gather support for the following attempt (ATTEMPT = TEST = TRY = EXPERIMENT): 

- Transform travis to use docker
- Run travis2docker (or a small modification of it) to create DEV docker files in repo
- Start a dev environment with docker build . && docker run and easily mount the working tree into it's place

Best advantage: Binaries for almost every platform are available in the docker ecosystem.
Manual maintenance can focus on the Travis files, the rest is already scripted...

@Moses: when talking about mounting instead of syncing via a network protocol via the docker volume command (read: let the docker host take care of this) we cannot AFAIK let the docker be the source, because docker gives precedence to the host files system and eventually deletes anything that was present in this path on the docker itself. So we might have a closer look on how we could include the git proxy workflow nevertheless. It's important though, that a local git can be easily referenced, so no internet connection is needed...

Thanks for your questions, if any!

Best, David

El vie., 16 oct. 2015 a las 11:58, David Arnold (<dar@devco.co>) escribió:
Let's turn to https://github.com/OCA/maintainer-tools/issues/146 for the tooling discussion, anyone who can moderate or better elaborate the facts, pros and cons, than I'm probably capable of: please be welcome!

Inputs from akretion and vauxoo are very promising, we need to decide on the right dosis of tooling for this specific purpose, and assure plattfrom independence.

El vie., 16 oct. 2015 a las 11:52, David Arnold (<dar@devco.co>) escribió:
Adapted.

El vie., 16 oct. 2015 a las 11:23, Daniel Reis (<dgreis@sapo.pt>) escribió:

I just took a closer look at it.
Words like "Recommended Editor" are notorious for triggering epic flame wars, such as emacs vs vim ;-)


Às 16:53 de 16/10/2015, David Arnold escreveu:
<blockquote cite="mid:CAOLEt-E1f1QPZht+0db==8EbPAxfX_4vyz7uTQxp1Po800UhEg@mail.gmail.com" type="cite">
Hi!

Based off on https://github.com/OCA/maintainer-tools/issues/146 and Daniel's suggestion, I made a recompilation here of recommended contributor tooling: https://github.com/OCA/maintainer-tools/wiki/Recommended-Contributor-Toolbox-(RCT)

It's WIP, but with your feedback, I'm commited to make it 1.0 at the indicated detail level in no time. I hope it might also trigger or revive connected discussions, such as repo layout. (https://github.com/OCA/maintainer-tools/issues/147) - For my taste, I invite you to discuss directly on the github issue to not bother the less involved.

I'll reference also a second mail to follow, with a proposal for modernizing informal communication with the help of gitter.im.

Best Regards

David

_______________________________________________

--

_______________________________________________

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe


--

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe




--



--

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe




--
Raphaël Valyi
Founder and consultant
+55 21 3942-2434


_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe