Posts tagged "technote"

• Major components:
  • A server/daemon which
    • manages docker objects, such as images, containers, network and data volumes
    • has a REST API
    • and a CLI (which uses the REST API)
  • A client which is the primary way to interact with Docker.
  • A registry (e.g. Docker Hub) that stores images
• App hierarchy:
  • Stack
  • Service: a container from a docker-compose file
  • Container:
    • When stopped: a runnable instance of an image
    • When started: a running image (i.e. image + state)
    • may be connected to a network and/or storage
    • can be stored as an image
  • Image:
    • A read-only template with instructions for creating a Docker container
    • an executable package
    • may be based on another image
• Files:
  • A Dockerfile defines an image
  • A docker-compose.yml defines one or more containers (a.k.a. services) that work together
• Tips:
  • You start a container (after it has been stopped) and run an image (which creates and starts a container).
  • ​Docker services can address each other through their container names as host name.

(src)

• list
  • other list
    • Numbered list

Some example from a text file.

Also available: VERSE, QUOTE and CENTER

(defun org-xor (a b)
  "Exclusive or."
  (if a (not b) b))

Use Ditaa for figures.

• Chain of immutable blocks (though hashes)
• Public, distributed replication (through peer to peer - p2p)
• Decentralized, trustless consensus (by collective self-interest)
• Forks are resolved through scoring (resulting in orphan blocks)

There are two ways to use Scrapy, the hard way and the quick-'n-dirty way:

• For the hard way you let Scrapy generate a project for you with the startproject command. This will give you all the bells and whistles you need for extensive web-scraping, including items, middleware and pipelines. This will allow you to write a whole nest of related (or unrelated) spiders and deploy them to the cloud. But for most use cases this is far more than is needed, and the easy way is sufficient.
• The easy way is to simply write a spider and run it with the runspider command. This will give you just one, simple spider, but I find that in most cases this is sufficient.

The scrapy shell command allows you the research the page and experiment with selectors.

The scrapy view command (both in the Scrapy shell and from the command line as a parameter to the scrapy command) opens a page in your browser, as seen by Scrapy. This prevents differences between how your browser GETs a page and how Scrape sees it.

The PyData Python Open Data Science Stack:

• numpy as np
  • axis=0 means columns and axis=1 means rows
• scipy
• sklearn
  • preprocessing
  • linear_model
  • cross_validation
  • confusion_matrix
  • svm
  • multiclass
• pandas as pd
  (The framework for data engineering, although others exist, like Bubbles.)
• bobobo for ETL

• binarization
• mean removal
• scaling
• normalization
• label encoding

Wikipedia Portal

Many consider Python 'the second best language for anything'. PHP may be the leading open source language for the web, Java may dominate the enterprise, R may be the dedicated language for data science, and bash may be the go to language for shell scripting, but in all these cases Python is at least a viable alternative.

What this means is that by learning Python you can be competitive in all these areas. And learning Python is relatively easy, because it is considered to be very beginner-friendly (while at the same time being powerful enough to intrigue the most advanced hackers).

But don't get carried away, because just learning Python (the programming language) is not enough. To be productive in any area, you will have to learn the appropriate package(s) as well.

The only area I can think of where Python is not at home, is the web client. There are client-side implementations like Skulpt, but these are still rather obscure. Javascript rulez (sic) in the browser. And with it, its elegant asynchronicity and use of callbacks. Both work in Python (the former since v3.4 through asyncio), but they are not as common.

Python was originally developed by Guide van Rossum who also was its BDFL for decades. But on July 12, 2018 he indicated he is going to step down.

Version 2.7 of Python has long been my default, but a couple of years ago I switched to version 3, which rectified some of the technical debt that had built up in version 2, at the cost of loosing backwards compatibility.

To describe something as clever is not considered a compliment in the Python culture. Alex Martelli - Python Cookbook (O’Reilly)

When working with Python, you will often encounter the phrase pythonic, indicating if something is 'proper python' or not. But what does this mean?

PEP20, or the Zen of Python, describes guiding principles for Python's design:

• Beautiful is better than ugly.
• Explicit is better than implicit.
• Simple is better than complex.
• Complex is better than complicated.
• Flat is better than nested.
• Sparse is better than dense.
• Readability counts.
• Special cases aren't special enough to break the rules.
• Although practicality beats purity.
• Errors should never pass silently.
• Unless explicitly silenced.
• In the face of ambiguity, refuse the temptation to guess.
• There should be one –and preferably only one– obvious way to do it.
• Although that way may not be obvious at first unless you're Dutch.
• Now is better than never.
• Although never is often better than right now.
• If the implementation is hard to explain, it's a bad idea.
• If the implementation is easy to explain, it may be a good idea.
• Namespaces are one honking great idea – let's do more of those!

Note that the Zen of Python uses terms like "is better than" and "beats", indicating that these are relative, rather than absolute, values. So, flat may be better than nested, but in specific conditions you may still use nesting.

PEP8 contains the Python style guide. Here you will find conventions about layout, whitespace, etc.

The Hitchhiker’s Guide to Python contains an extensive section on writing great code.

To me, constructs like list comprehension and generators feel very Python-specific, although they also occur in other languages.

Pipenv, as its name suggest, combines pip and virtualenv:

Pipenv is primarily meant to provide users and developers of applications with an easy method to setup a working environment. It harnesses Pipfile, pip, and virtualenv into one single command. It features very pretty terminal colors.

Pipenv simplified my Python workflow significantly. Instead of creating a virtual environment, activating it, running pip install, etc., I just do pipenv shell and I'm in business.

Here's a list of the packages I use most:

• Both flask and Django for web development, on the client combined with Bootstrap, Datatables and Chart.js.
• Pandas for data analysis.
• Scrapy for web scraping.

• Nikola for blogging on Github pages.

  But remember, this are just my preferences, YMMV. If you want to dig up your own favorites, head over to the curated list of Python packages at Awesome Python.

• For REPL there is no standard like ipython, but in some cases I prefer ptpython.
• The command python -m site

• bobobo
• Doit

I sometimes blog with Markdown-files and publish it with static site generator Nikola. While writing I can check the live result with the nikola auto command.

One of the lesser known features of Nikola are shortcodes: simple snippets that you can use throughout your blog. They come in the varieties built-in, community-provided, and home-made. The simplest way to roll your own is by using templates in the shortcodes directory. For example, I have defined a shortcodes/attention.tmpl that gives me {{% attention %}} whenever I want it.

Good software [Beck]:

1. Passes the tests
2. Reveals intention
   • I will contend that Conceptual Integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas. [Brooks]
   • Simplicity and clarity —in short: what mathematicians call "elegance"— are not a dispensable luxury, but a crucial matter that decides between success and failure [Dijkstra]
   • Convention over configuration
   • Separation of Concerns
   • Maximum coherence, minimum dependency
3. No duplication
   • Don’t Repeat Yourself (DRY)
   • Single Point of Truth (SPOT)
4. Fewest elements
   • A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away. [Antoine de Saint-Exupéry]
   • Less is More
   • Occam's razor: the simplest option is usually correct
   • Keep It Small and Simple (KISS) until proven complex
   • Make everything as simple as possible, but not simpler [Einstein]
   • Measuring programming progress by lines of code is like measuring aircraft building progress by weight. [Gates]

Snippets:

• From the Gang of Four:
  • Program to an interface, not an implementation
  • Favor object composition over class inheritance
• Algemene namen:
  • Foo, bar
  • Alice, Bob, Carol & Dave
• Make the change easy. Then make the easy change.
• Technology develops from the primitive via the complex to the simple. [Antoine de Saint-Exupéry]
• Command Query Separation: Functions that change state should not return values and functions that return values should not change state.
• Death by Arguments
• Refactoring Catalog
• Principles of OOD:
  • Class design:
    • SRP (Single Responsibility Principle): A class should have one, and only one, reason to change.
    • OCP (Open Closed Principle): You should be able to extend a classes behavior, without modifying it.
    • LSP (Liskov Substitution Principle): Derived classes must be substitutable for their base classes.
    • ISP (Interface Segregation Principle): Make fine grained interfaces that are client specific.
    • DIP (Dependency Inversion Principle): Depend on abstractions, not on concretions.
  • Package¹ cohesion (what to put inside packages):
    • REP (Release Reuse Equivalency Principle): The granule of reuse is the granule of release.
    • CCP (Common Closure Principle): Classes that change together are packaged together.
    • CRP (Common Reuse Principle): Classes that are used together are packaged together.
  • package couplings (metrics that evaluate the package structure of a system):
    • ADP (Acyclic Dependencies Principle): The dependency graph of packages must have no cycles.
    • SDP (Stable Dependencies Principle): Depend in the direction of stability.
    • SAP (Stable Abstractions Principle): Abstractness increases with stability.

About IT:

• IT connects people and systems
• IT is a craft
• IT is human labour

Snippets:

• Deploy Early and Often (DEO) / Release Early, Release Often (RERO)
• Problem vs Work
• Realistische ambitie
• Pareto principle: 20/80% rule
• Lacking quality, rules abound.

The mindmap below shows how I understand Meteor:

Meteor mindmap

The client handles routing and rendering. Routing is taken care of by Flow router, which uses Blaze as template engine (Angular and React are also supported). Templates are divided in three layers:

• layouts define the general structure of your site, using pages
• pages are smart, which means they collect data and feed them to components
• components do not interact with anything except though parameters, which makes them highly reusable

Templates can be controlled through helpers and event handlers (including onCreated).

The server uses Mongodb as a datastore. Data is stored in collections, which are basically persistent JSON documents. A schema language is used to define the datastructures, so data can be validated. Schema also drive the autoform package. Collections can be controlled through helpers and hooks.

The client and server communicate through a publish/subscribe mechanism and through methods. The server controls data access by selectively publishing data to the client. The client (pages) collect data through subscriptions. Methods are Meteor's remote procedure calls (RPCs) and can use schema for data validation.

Some final tips & tricks:

• waitOn (package Iron Router) lets you defer execution untill a subscription has finished
• Meteor methods <i>can</i> be called synchronously on the server, but <i>must</i> be called asynchronously on the client
• Global variables aren't available from templates. Access them with template helper.
• add c:\Windows\System32 to path on tasklist.exe error
• The autorun function lets you define a function that is run automatically when a reactive data source changes.

Just like all other modern frameworks, Meteor relies heavily on third party packages from its thriving eco-system. The Meteor Guide gives an opinionated overview of which packages to use. The table below shows these packages in the middle column, along with some additional packages I prefer in the final column. For more packages visit Atmosphere.js. You can import all these package definitions by importing the file packages from this repository into the file packages of your project and the remove any package you don't need.

<tr><th>subject</th><th>guide</th><th>extra</th></tr>