1. Mailing Lists
  2. Contributors
  3. Re: Recommended Contributor Tooling (RCT)

Contributors mailing list archives

contributors@odoo-community.org

Browse archives

Re: Recommended Contributor Tooling (RCT)

Re: Recommended Contributor Tooling (RCT)

Avatar

Re: Recommended Contributor Tooling (RCT)

by
dar@devco.co
- 19/10/2015 02:21:22
Thank you Raphael for your further details, this get's really interesting and productive :D Thank you!

Here are my comments:

basicallly what sucks in Odoo is the the repo is huge. So a large part of the Voodoo Python launcher is made to create the options of the docker run command, specially which folder you'll mount to share your Odoo repo (Odoo is too much of a buggy moving target to be bundled inside the image and it's to heavy to be downloaded always).
docker mounting as of today on windows hosts is a mess. Virtual box is buggy, docker-machine improving. But there is no real good way to get the sources mounted into the docker. I'm working with rsync right now as the next best option, but this obviously consumes a lot of the docker's in-memory filesystem, which I think is the problem you are referring to.
gladly there exists docker volume drivers for nfs and even gitRepos (!!), and now with the release of the repsective api in 1.9 we can count on to get possibilities multiplied. So best strategy here would be to use some network protocol based volume driver to communicate with the local host and mount the source code. This can then be adapted, according to the workflow to be a remote git, a local git a local folder, a network storage, a shared folder (here is the sharing) or whatever...


Alternatively I thought about putting the Voodoo inteligence inside the portable Linux image and like running it twice: one to first output the run options and then eval the command of the second run. That's e technique used by Consul https://github.com/gliderlabs/docker-consul/tree/legacy#runner-command but again sadly that wouldn't work on Windows.
That's pretty much what I mean by Docker Inception technique. :) But as mentioned above, we could be still more elegant with the priviledge flag. We then port forward the last docker to it's parent docker, and port forward the parent docker to the docker daemon. This way they should be able to span a common comunication plan and linking should be possible (however I've to investigate on this further).

Voodoo also have a few other useful commands. We often use "voodoo new project_name" to bootstrap a new project and hack on something.

Overall, I think the Voodoo Python launcher is useful. But what I think is we can have 80% of the useful features in just 20 lines of Ruby in a Vagrantfile. Then "voodoo run" would be just "Vagrant up" on windows and sharing a project would just work (not the advanced Docker compose feature but f*ck Windows right...)
Wouldn't it be greate to split out those bootstrapping features and integrate it more closely with savoir's protoype and the odoo scaffold command itself? This could by just so f*cking elegantly made available through docker inception technique, too :D In theory, there is not even reworking needed, we just can construct a docker that exposes those bootstrapping via insepction technique... NICE!

Best Regards, and thanks! This is getting cool and fun!



El dom., 18 oct. 2015 a las 20:09, David Arnold (<dar@devco.co>) escribió:
Raphael, thanks a lot for your detailed explication!

This gives me the possibility to ask some more detailed questions, I'm interested in:

Voodoo is basically a pre-built dev environment bundled a single Docker image. We think it's the right mileage between the pre-build-dependencies for a turn key startup and the flexibility of Anybox recipe to deal with any module branch and Odoo branch, just like using the recipe without Docker. 
I wanted to open up discussion on this point because of the following reason: travisfile2dockerfile by vauxoo permits using the travisfiles, that since the implementation of travis by oca are present (!) and maintained (!) in every repo. travisfile2dockerfile adds therefor a dynamic environmentcreation, which is the same goal what you want to achieve with anybox, if I understadn you right. On the other hand, and this is the elegance I was aiming for, Moses probably can easily rewrite his t2d library to work as a maintenance script, that automatically transforms and updates (via PR and Github Bot Account) each repo's individual "last mile dockerfile" (based on a common oca base image). This way, I can easily implement and transform the docker contribution guide and adapt it to the oca ecosystem. The only host dependency is indeed git and docker, this way. (and docker-machine abstraction handler on OSX and Windows).

It makes projects very lightweight, self-contained (even the dev database is inside the project folder, just like Rails with sqlite) and extremely easy to share between developers. And we also have easy tools to push such a dev env to the cloud in just one command.
I assume this command is the generic docker export / docker import or some lightweight scripting and scp streaming arround it? Or can be acomplished that way? As I understand you, you simply persist the database volume onto the host for direct acces? This might be a good and desirable idea, and is supposedly accomplished by docker -v command, right? I take note of that, to include it into the contribution guideline, I'm aiming at. 

The python package of Voodoo is just a handy launcher. initially it was 20 lines of bash but then my co-workers rewrote it as a Python wrapper around Docker Compose, giving it unlimited flexibility to attach custom micro-services such as a mail catcher, a pre-confgured Eclipse, a database browser, a Solr server, a Rails e-commerce, anything you want, specified in the same project repo.
That is a very noble intention. I would however point out, that it is important to not depend on any local host dependency. But that wouldn't probably be a problem, you can easily rewrite that and include it into the docker host itself. This is called docker inception technique. It would feel something like docker run --privileged aktretion/voodoo commandxyz. Neither bash nor python are good enough for the local host, because of portability. But I feel we do not need any tooling, one git and one docker command is completely enough (just 2 standard BINARY commands) To setup the perfect dev environment. Can those microservices not be achieved by a standard docker-compose definition? We could move the "necesary" to a docker in docker "privileged" architecture, such as db, debugging and the odoo environment and leave optional microservices to an still optional docker-compose (like this windows user are not doomed).

Now yes, this depending on the maturity level of Docker compose on Windows and Mac, the launcher may not work on these platforms. Well in that case, in a few hours of work we can ressucitate the Vagrantfile launcher that will mimic the 80% of important features to fire the Docker workspace container on these plateforms.
I'm actually against this approach, because it add's even more unecessary plumbing and tooling to the local host. I'm very against this, because it increases setup times. Actually I have in my mind a design, where we would not need docker-compose, so this would be elegantly resolved by "reduce to the max" strategy.

I believe this is what you wanted to achieve. And yes it's mature, we have been more than a dozen of people using it for 6 moths now at Akretion...
That is definitely a valuable argument, however, if we could reach out to completely cut out this layer at all, which needs stabilization, why shouldn't we?

Best Regards, David

El dom., 18 oct. 2015 a las 19:38, Raphaël Valyi (<rvalyi@akretion.com>) escribió:
David,

Voodoo is basically a pre-built dev environment bundled a single Docker image. We think it's the right mileage between the pre-build-dependencies for a turn key startup and the flexibility of Anybox recipe to deal with any module branch and Odoo branch, just like using the recipe without Docker. It makes projects very lightweight, self-contained (even the dev database is inside the project folder, just like Rails with sqlite) and extremely easy to share between developers. And we also have easy tools to push such a dev env to the cloud in just one command.

The python package of Voodoo is just a handy launcher. initially it was 20 lines of bash but then my co-workers rewrote it as a Python wrapper around Docker Compose, giving it unlimited flexibility to attach custom micro-services such as a mail catcher, a pre-confgured Eclipse, a database browser, a Solr server, a Rails e-commerce, anything you want, specified in the same project repo.

Now yes, this depending on the maturity level of Docker compose on Windows and Mac, the launcher may not work on these platforms. Well in that case, in a few hours of work we can ressucitate the Vagrantfile launcher that will mimic the 80% of important features to fire the Docker workspace container on these plateforms.

Imagine that Voodoo is just Devstep for Odoo:

I believe this is what you wanted to achieve. And yes it's mature, we have been more than a dozen of people using it for 6 moths now at Akretion...

Cheers.

On Sun, Oct 18, 2015 at 5:38 PM, David Arnold <dar@devco.co> wrote:
Example, according to documentation on an XYZ python library:

$ python setup.py develop
Traceback (most recent call last):
  File "setup.py", line 11, in <module>
    from setuptools import find_packages
ImportError: No module named setuptools

Sorry, you just don't want to burden this onto someone who just want's to get started. Even to us it might be an obvious error. It's not the user's fault, that this is not working. It is the programmer's (as almost always) ...
El dom., 18 oct. 2015 a las 14:24, David Arnold (<dar@devco.co>) escribió:
Ps.: If you can ship me a windows 64 compiled binary, I'm happy to evaluate it thoroughly!

El dom., 18 oct. 2015 a las 14:22, David Arnold (<dar@devco.co>) escribió:
Hi Raphael,

thanks for your feedback. Indeed it is possible. Actually I'm still waiting more detailed technical description of voodoo to be able to draw my own judgment. A statement  "will be exactly what you want" assumes that you "understood exactly what I want". What I saw from a quick look it is a python library. So unfortunately this is not a common enough ground for tooling, mainly because it leaves higher level tooling (like python and virtualenv) to the user. This is not nice and not even necessary. Please correct my, though, if I understood the functioning of voodoo wrong. Again, I would be delighted by a short technical description comprising at least a description of what issues are addressed by what strategy. That would be really helpful.

Thanks and best regards, David

El dom., 18 oct. 2015 a las 13:08, Raphaël Valyi (<rvalyi@akretion.com>) escribió:
David,

sorry but I really think if you help contributing a Vagrantfile launcher for Voodoo it will be exactly what you want and following best field practices...

Without help we will make that Vagrantfile (unless Docker compose run smooth on Windows) but help could speed this up.

Regards.

On Sun, Oct 18, 2015 at 5:23 AM, David Arnold <dar@devco.co> wrote:
Moses, we have the problem, that most windows user, that install git via he official installer will opt for the (recommended) option that files are checked out with CRLF. I have the gut feeling, that if they have done so, bindmounting those files will produce non understandable errors in the outside layers of their stack for them. This would be another argument for having the git proxy and working over a network on the docker filesystem. Or we populate respective sources into bindmounted directories AFTER the startup in the run script... I mean to be able to leverage docker volume and OS standard tooling instead of having to especially prepare the IDE. And keep that stuff out of the odoo environment.

El dom., 18 oct. 2015 a las 1:47, David Arnold (<dar@devco.co>) escribió:
Hi Eric, thank you for your feedback!

What I want to achieve is a standard way for new contributors to get coding quickly, weather they be on windows, osx or linux. My role model newcomer does not know python, nor git, nor does he know other stuff. He knows barely some basic concepts of the command line, and still he is valuable, because he might be able to copy paste some code and get his business case working. And, he might grow and evolve...

Installing python is cumbersome and litters your computer, and after you do a computer reset (I do that every 2 months probably) you have to start again setting everything up. Setup virtualenv is even more cumbersome, actually I never got a hold on it. So be python based solution discarted. We need binaries to get people going. Enter docker, it ships binaries. But I'm not getting at deploying. I'm aiming at the perfect, most cross platform, easy, dead simple, get going in less than 5 seconds development stack.

The point in the above mail is derived from Moses's (i feel) briliant idea to reuse the maintainance work, that is already present in the travis files. He wrote a tool, that transforms it into a buildable docker file (https://github.com/vauxoo/travis2docker). This set's up a complete environment (python deps, odoo deps other deps). So if you want to start hacking on some oca repo, you just have to clone it and build the automatically maintained dockerfile which includes the specific environment of this specific repo. And maintainers would carry on maintaining the travis files only, which have kind of the historic precedence rights on their side.

After building and docker run, you can hapily modify the src on your host and everthing get's bindmounted or synced into the docker, so you go to localhost:8069 and see what you have done instantly (with pynitify and odoo autoreload).

That's the kind of instantaneousness I want to see, also on windows systems. And I have not mingled one single time with anything of the kind of virtualenv, pythondeps, git blabla, missed module dependencies. It's just something like git clone && docker build . && docker run. Required binaries: git, docker. That's all - available for nearly any thinkable of common platform. And you are good to go hacking on your working directory.

Not: This is quite agnostic of whatever (probably git based) upper continuous integration workflow you chose. But this is anyhow not a concern of the one how just wants to get his hands dirty.

I'm trying to collaboratively discuss the best design for such a tool and am committed to come up with a beta-worthy solution for testing. Maybe I need some help here and there, but with communities support, I think it's doable.

Best, David

El sáb., 17 oct. 2015 a las 22:08, Eric Caudal (<eric.caudal@elico-corp.com>) escribió:

Hi David,
We appreciate your  various contribution but in this case, I am not sure to understand where you are getting at.
Docker is very nice environment and I am in favor for the deployment although the techno is style young is not completely bullet proof.
We have our own here based on OCB and refreshed  every week (as of next week) : https://hub.docker.com/u/elicocorp/.
You can grab docker-files here:https://github.com/Elico-Corp/odoo-docker
(I will send a separate mail anyway to check if community could be interested in maintaining this repo in OCA)

But I dont see the point of your exercise. Current infrastructure is solid for travis: once your code is solid and your docker file is well structured, there is no need to travis the dockers (IMHO).

Resources are limited and we do not want to tumble our infrastructure that we have needed month to stabilize so we need clear objective and pros for the community and you will come with a fully prepared solution.
We barely have resources for maintainance and cannot really just TRY things...
If you can come with solid project definition and proof of concept + some resources to maintain it in the community, we could consider it

--
Eric Caudal [Founder and CEO]
Skype: elico.corp. Phone: + 86 186 2136 1670 (Cell), + 86 21 6211 8017/27/37 (Office)
Elico Shanghai (Shenzhen/Singapore) Odoo Gold Partner, best Odoo Partner 2014 for APAC
On 10/18/2015 10:37 AM, David Arnold wrote:
<blockquote cite="mid:CAOLEt-EMOGXd9JjKassUDD6zkzOD5ZFfsgcop13qFMD1TpJM9Q@mail.gmail.com" type="cite">
Having talked with Moses (vauxoo) bilateral about his solution further, I would like to gather support for the following attempt (ATTEMPT = TEST = TRY = EXPERIMENT): 

- Transform travis to use docker
- Run travis2docker (or a small modification of it) to create DEV docker files in repo
- Start a dev environment with docker build . && docker run and easily mount the working tree into it's place

Best advantage: Binaries for almost every platform are available in the docker ecosystem.
Manual maintenance can focus on the Travis files, the rest is already scripted...

@Moses: when talking about mounting instead of syncing via a network protocol via the docker volume command (read: let the docker host take care of this) we cannot AFAIK let the docker be the source, because docker gives precedence to the host files system and eventually deletes anything that was present in this path on the docker itself. So we might have a closer look on how we could include the git proxy workflow nevertheless. It's important though, that a local git can be easily referenced, so no internet connection is needed...

Thanks for your questions, if any!

Best, David

El vie., 16 oct. 2015 a las 11:58, David Arnold (<dar@devco.co>) escribió:
Let's turn to https://github.com/OCA/maintainer-tools/issues/146 for the tooling discussion, anyone who can moderate or better elaborate the facts, pros and cons, than I'm probably capable of: please be welcome!

Inputs from akretion and vauxoo are very promising, we need to decide on the right dosis of tooling for this specific purpose, and assure plattfrom independence.

El vie., 16 oct. 2015 a las 11:52, David Arnold (<dar@devco.co>) escribió:
Adapted.

El vie., 16 oct. 2015 a las 11:23, Daniel Reis (<dgreis@sapo.pt>) escribió:

I just took a closer look at it.
Words like "Recommended Editor" are notorious for triggering epic flame wars, such as emacs vs vim ;-)


Às 16:53 de 16/10/2015, David Arnold escreveu:
<blockquote cite="mid:CAOLEt-E1f1QPZht+0db==8EbPAxfX_4vyz7uTQxp1Po800UhEg@mail.gmail.com" type="cite">
Hi!

Based off on https://github.com/OCA/maintainer-tools/issues/146 and Daniel's suggestion, I made a recompilation here of recommended contributor tooling: https://github.com/OCA/maintainer-tools/wiki/Recommended-Contributor-Toolbox-(RCT)

It's WIP, but with your feedback, I'm commited to make it 1.0 at the indicated detail level in no time. I hope it might also trigger or revive connected discussions, such as repo layout. (https://github.com/OCA/maintainer-tools/issues/147) - For my taste, I invite you to discuss directly on the github issue to not bother the less involved.

I'll reference also a second mail to follow, with a proposal for modernizing informal communication with the help of gitter.im.

Best Regards

David

_______________________________________________


_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe


_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe

--

_______________________________________________

_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe


--
Raphaël Valyi
Founder and consultant
+55 21 3942-2434


_______________________________________________
Mailing-List: http://odoo-community.org/groups/contributors-15
Post to: mailto:contributors@odoo-community.org
Unsubscribe: http://odoo-community.org/groups?unsubscribe

Reference