This project focuses on understanding the language ecosystem, not getting into programming details.

🌄 Python's Habitat

This topic describe how to set up the environment to Python developement.

• Preparing the Environment for the Python
• Check Python Configuration
• Advanced settings of Python
• What is a virtual environment and how it works

🐍 Python's Taxonomy

This topic describe features of the pattern of Python projects.

• Package manager
• Requirements file
• Deterministic build

💢 Python's Behavior

This topic describe how the language is designed and how it works.

• Compiler and interpreter
• How Python runs a program
• How Python search path module
• How Python manages process and threads
• How Python manages memory
• How to deeply understand Python code execution (debug)

🐛 Python's Feeding

This topic describes formatting patterns following a style guide.

• Linters
• Knobs (Google YAPF)
• Identation and Length
• Line Break After a Binary Operator
• Naming
• Encoding
• Strings ' ' and " "
• Comments #
• Imports
• Dunders to Documentation
• Annotation Functions
• Type Hints
• String Concatenation
• String Methods
• Exception
• Return
• Type Comparisons
• Methods with numerous parameters
• Docstrings

🔍 Python's Other Features

Extra topics to see.

Linux

Python needs a set of tools that are system requirements. If necessary, install these requirements with this command:

sudo apt update sudo apt install\ software-properties-common\ build-essential\ libffi-dev\ python3-pip\ python3-dev\ python3-venv\ python3-setuptools\ python3-pkg-resources

Now, the environment is done to install Python

sudo apt install python

Windows

On Windows, I recommend using the package manager chocolatey and set your Powershell to can work as admin. See this tutorial.

Now, install Python

choco install python 

Test

python --version 

Watch

python --version

Watch

which python

Watch

sudo update-alternatives --list python

alias python=python3

Python can run in a virtual environment with isolation from the system.

Arquitecture of Execution

Virtualenv enables us to create multiple Python environments which are isolated from the global Python environment as well as from each other.

When Python is initiating, it analyzes the path of its binary. In a virtual environment, it's actually just a copy or Symbolic link to your system's Python binary. Next, set the sys.prefix location which is used to locate the site-packages (third party packages/libraries)

Symbolic link

• sys.prefix points to the virtual environment directory.
• sys.base.prefix points to the non-virtual environment.

Folder of virtual environment

ll # random.py -> /usr/lib/python3.6/random.py # reprlib.py -> /usr/lib/python3.6/reprlib.py # re.py -> /usr/lib/python3.6/re.py # ...

tree ├── bin │ ├── activate │ ├── activate.csh │ ├── activate.fish │ ├── easy_install │ ├── easy_install-3.8 │ ├── pip │ ├── pip3 │ ├── pip3.8 │ ├── python -> python3.8 │ ├── python3 -> python3.8 │ └── python3.8 -> /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 ├── include ├── lib │ └── python3.8 │ └── site-packages └── pyvenv.cfg

Create Virtual Environment

Watch

Create virtual environment

virtualenv -p python3 <NAME_ENVIRONMENT>

Activate

source <NAME_ENVIRONMENT>/bin/activate

# Pipfile [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] [packages] requests = "*" numpy = "==1.18.1" pandas = "==1.0.1" wget = "==3.2" [requires] python_version = "3.8" platform_system = 'Linux'

# requirements.txt requests matplotlib==3.1.3 numpy==1.18.1 pandas==1.0.1 wget==3.2

pip3 install --user pipenv

Requirements.txt is file containing a list of items to be installed using pip install.

pip3 freeze

pip3 freeze > requirements.txt

cat requirements.txt

pip3 install -r requirements.txt

Using pip and requirements.txt file, have a real issue here is that the build isn’t deterministic. What I mean by that is, given the same input (the requirements.txt file), pip does not always produce the same environment.

A set of command line tools to help you keep your pip-based packages fresh.

• Distinguish direct dependencies and versions
• Freeze a set of exact packages and versions that we know work
• Make it reasonably easy to update packages
• Take advantage of pip's hash checking to give a little more confidence that packages haven't been modified (DNS attack)
• Stable

pip install pip-tools 

pip3 freeze > requirements.in

pip-compile --generate-hashes requirements.in

pip-compile --generate-hashes requirements.in

CPython can be defined as both an interpreter and a compiler.

• The compiler converts the .py source file into a .pyc bytecode for the Python virtual machine.
• The interpreter executes this bytecode on the virtual machine.

The principal feature of CPython, is that it makes use of a global interpreter lock (GIL). This is a mechanism used in computer-language interpreters to synchronize the execution of threads so that only one native thread can execute at a time.
Therefore, for a CPU-bound task in Python, single-process multi-thread Python program would not improve the performance. However, this does not mean multi-thread is useless in Python. For a I/O-bound task in Python, multi-thread could be used to improve the program performance.

Multithreading in Python

The Python has multithreads despite the GIL. Using Python threading, we are able to make better use of the CPU sitting idle when waiting for the I/O bound, how memory I/O, hard drive I/O, network I/O.

This can happen when multiple threads are servicing separate clients. One thread may be waiting for a client to reply, and another may be waiting for a database query to execute, while the third thread is actually processing Python code or other example is read multiples images from disk.

NOTE: we would have to be careful and use locks when necessary. Lock and unlock make sure that only one thread could write to memory at one time, but this will also introduce some overhead.

1. Tokenize the source code: Parser/tokenizer.c
2. Parse the stream of tokens into an Abstract Syntax Tree (AST): Parser/parser.c
3. Transform AST into a Control Flow Graph: Python/compile.c
4. Emit bytecode based on the Control Flow Graph: Python/compile.c

When Python executes this statement:

import my_lib

The interpreter searches my_lib.py a list of directories assembled from the following sources:

• Current directory
• The list of directories contained in the PYTHONPATH environment variable
• In directory which Python was is installed. E.g.

The resulting search can be accessed using the sys module:

import sys sys.paths # ['', '/usr/lib/python38.zip',  # '/usr/lib/python3.8', # '/usr/lib/python3.8/lib-dynload', # '/home/campos/.local/lib/#python3.8/site-packages', # '/usr/local/lib/python3.8/dist-packages', # '/usr/lib/python3/dist-packages']

Now, to see where a packeage was imported from you can use the attribute __file__:

import zipp zipp.__file__ # '/usr/lib/python3/dist-packages/zipp.py'

NOTE: you can see that the __file__ directory is in the list of directories searched by the interpreter.

TODO

TODO

"Readability counts"

• 4 spaces
• Limit all lines to a maximum 72 characteres to docstring or comments
• Limit all lines to a maximum 79 characteres to code

# Aligned with opening delimiter. foo = long_function_name(var_one=0.0, var_two=0.0, var_three=0.0, var_four=0.0)

• Class Name (camelCase): CapWords()
• Variables (snack_case): cat_words
• Constants: MAX_OVERFLOW

income = (gross_wages + taxable_interest + (dividends - qualified_dividends) - ira_deduction - student_loan_interest)

By default: UTF-8

# -*- coding: UTF-8 -*- <code>

Single quotation marks and strings with double quotation marks are the same.

• First word need upper case.
• Comments in-line separete by 2 spaces.

x = x + 1 # Compensar borda

Following order:

1. Standard library imports.
2. Related third party imports. (parte de terceiros)
3. Local application/library specific imports.

import argparse import configparser import os import mysql.connector import my_module

Yes:

import os import sys

No:

import os, sys

No problems:

from subprocess import Popen, PIPE

__version__ = '0.1' __author__ = 'Bruno Campos'

• Use ''.join(), to concatenate 3 or more:

os.path.dirname.join(stringA + stringB + stringC + stringD)

• This optimization is fragile even in CPython. Not use:

stringA + stringB + stringC + stringD

• Use string methods instead of the string module because, String methods are always much faster.
• Use ''.startswith() and ''.endswith() instead of string slicing to check for prefixes or suffixes.

Yes: if foo.startswith('bar'): No: if foo[:3] == 'bar':

Limit the clausule try: minimal code necessary.

Yes:

try: value = collection[key] except KeyError: return key_not_found(key) else: return handle_value(value)

No:

try: # Too broad! return handle_value(collection[key]) except KeyError: # Will also catch KeyError raised by handle_value() return key_not_found(key)

• Objetivo de responder à pergunta "O que deu errado?" programaticamente, em vez de apenas afirmar que "Ocorreu um problema"

"Should explicitly state this as return None"

• Be consistent in return statements.
• Todas as instruções de retorno em uma função devem retornar uma expressão ou nenhuma delas deve.

Yes:

def foo(x): if x >= 0: return math.sqrt(x) else: return None

No:

def foo(x): if x >= 0: return math.sqrt(x)

• Always use isinstance()

Yes: if isinstance(obj, int): No: if type(obj) is type(1):

"Don’t use comments to specify a type, when you can use type annotation."

• Atua como um linter (analisador de código para mostrar erros) muito poderoso.
• O Python não atribui nenhum significado a essas anotações.
• Examples:

Method arguments and return values

def func(a: int) -> List[int]:

def hello_name(name: str) -> str: return (f'Hello' {name}')

Declare the type of a variable (type hints)

a = SomeFunc() # type: SomeType

Isso informa que o tipo esperado do argumento de nome é str . Analogicamente, o tipo de retorno esperado é str .

def send_email(address, # type: Union[str, List[str]] sender, # type: str cc, # type: Optional[List[str]] bcc, # type: Optional[List[str]] subject='', body=None # type: List[str] ): """Send an email message. Return True if successful.""" <code>

TODO

• https://docs.python.org/3/library/typing.html#module-typing

• https://medium.com/@shamir.stav_83310/the-other-great-benefit-of-python-type-annotations-896c7d077c6b
• https://www.python.org/dev/peps/pep-0484/
• https://blog.jetbrains.com/pycharm/2015/11/python-3-5-type-hinting-in-pycharm-5/

• Docstrings must have:
  • Args
  • Returns
  • Raises

Simple Example

def say_hello(name): """  A simple function that says hello...  Richie style  """ print(f"Hello {name}, is it me you're looking for?")

Example partner Google

def fetch_bigtable_rows(big_table, keys, other_silly_variable=None): """Fetches rows from a Bigtable.   Retrieves rows pertaining to the given keys from the Table instance  represented by big_table. Silly things may happen if  other_silly_variable is not None.   Args:  big_table: An open Bigtable Table instance.  keys: A sequence of strings representing the key of each table row  to fetch.  other_silly_variable: Another optional variable, that has a much  longer name than the other args, and which does nothing.   Returns:  A dict mapping keys to the corresponding table row data  fetched. Each row is represented as a tuple of strings. For  example:   {'Serak': ('Rigel VII', 'Preparer'),  'Zim': ('Irk', 'Invader'),  'Lrrr': ('Omicron Persei 8', 'Emperor')}   If a key from the keys argument is missing from the dictionary,  then that row was not found in the table.   Raises:  IOError: An error occurred accessing the bigtable.Table object.  """ return None

Such a docstring becomes the __doc__ special attribute of that object.

• Simple Example

print(say_hello.__doc__) # A simple function that says hello... Richie style

• Example partner Google

• Create manual: man
• Is a built-in function help() that prints out the objects docstring.

>>> help(say_hello) Help on function say_hello in module __main__: # say_hello(name) # A simple function that says hello... Richie style

• Docstrings must show how to use script
• Must doc:
  • Usage: sintax command line
  • Examples
  • Arguments required and optional

""" Example of program with many options using docopt. Usage:  options_example.py [-hvqrf FILE PATH]  my_program tcp <host> <port> [--timeout=<seconds>]  Examples:  calculator_example.py 1 + 2 + 3 + 4 + 5  calculator_example.py 1 + 2 '*' 3 / 4 - 5 # note quotes around '*'  calculator_example.py sum 10 , 20 , 30 , 40  Arguments:  FILE input file  PATH out directory  Options:  -h --help show this help message and exit  --version show version and exit  -v --verbose print status messages  -q --quiet quiet mode  -f --force  -t, --timeout TIMEOUT set timeout TIMEOUT seconds  -a, --all List everything.  """ from docopt import docopt if __name__ == '__main__': arguments = docopt(__doc__, version='1.0.0rc2') print(arguments)

A docstring to a function or method must resume:

• behavior
• arguments required
• arguments optional
• default value of arguments
• returns
• raise Exceptions

Example

def says(self, sound=None): """Prints what the animals name is and what sound it makes.   If the argument `sound` isn't passed in, the default Animal  sound is used.   Parameters  ----------  sound : str, optional  The sound the animal makes (default is None)   Raises  ------  NotImplementedError  If no sound is set for the animal or passed in as a parameter.  """ if self.sound is None and sound is None: raise NotImplementedError("Silent Animals are not supported!") out_sound = self.sound if sound is None else sound print(self.says_str.format(name=self.name, sound=out_sound))

A docstring para uma classe deve resumir seu comportamento e listar os métodos públicos e variáveis ​​de instância. Se a classe se destina a ser uma subclasse e possui uma interface adicional para subclasses, essa interface deve ser listada separadamente (no docstring). O construtor de classe deve ser documentado na docstring para seu método init . Os métodos individuais devem ser documentados por seus próprios docstring.

Example

class SimpleClass: """Class docstrings go here.""" def say_hello(self, name: str): """Class method docstrings go here.""" print(f'Hello {name}')

Class docstrings should contain the following information:

• A brief summary of its purpose and behavior
• Any public methods, along with a brief description
• Any class properties (attributes)
• Anything related to the interface for subclassers, if the class is intended to be subclassed

• PEP 08
• PEP 484
• PEP 257
• https://realpython.com/python-pep8/#naming-conventions
• https://pep8.org
• Style guide Google: https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings

Methods with numerous parameters are a challenge to maintain, especially if most of them share the same datatype.
These situations usually denote the need for new objects to wrap the numerous parameters.

• too many arguments

def add_person(birthYear: int, birthMonth: int, birthDate: int, height: int, weight: int, ssn: int): '''too many arguments''' . . .

• preferred approach

def add_person(birthdate: 'Date', measurements: 'BodyMeasurements', ssn: int): '''preferred approach''' . . .

cyclomatic complexity counts the number of decision points in a method

Interview Questions on Virtual Environment

1. What is virtual environment in Python?
2. How to create and use a virtual environment in Python?
3. How do Python virtual environments work?