Anselmos Blog

Web based terminal emulators

2018-04-08T22:00:00+02:00

Let's find out what web based terminal emulator there are.

In this article I'm only going to guide you to the possible emulators.

Each of the emulator will have it's own article.

To The Point

The list

This is the list of Web based terminal emulators that I've found:

Antonylesuisse QWeb - [How to use QWeb]
ShellInABox - [How to use ShellInABox]
AnyTerm - [AnyTerm web based terminal emulator]
GateOne - [How to use GateOne emulator]
tty.js - [Using TTY.js as an web-based terminal emulator]
Gotty - [Using gotty to share terminal emulator on web]
Tmate.io - [Making your own tmux tmate server]
Wetty - [Using Wetty as an alternative to ajax/any-term]
ttyd - [Creating ttyd server to share terminal on web]
Termshare - [Using Termshare terminal sharing server]
Mosh - [Using Mosh as an ssh alternative for mobile terminal]

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

PYTHONSTARTUP What it is and How to use it

2018-04-07T23:05:00+02:00

Let's find out what PYTHONSTARTUP is and how can we use it.

To The Point

According to documentation:

PYTHONSTARTUP

If this is the name of a readable file, the Python commands in that file are executed before the first prompt is displayed in interactive mode. The file is executed in the same namespace where interactive commands are executed so that objects defined or imported in it can be used without qualification in the interactive session. You can also change the prompts sys.ps1 and sys.ps2 and the hook sys.interactivehook in this file.

To put it into perspective, let's check how to use it.

Let's create a .pythonrc file with this:

print("how_use_pythonstartup")

Then lets setup PYTHONSTARTUP with this bash command:

export PYTHONSTARTUP=~/.pythonrc

Now let's call python interactive mode with python, and you will find something similar to this:

Python 2.7.6 (default, Nov 23 2017, 15:49:48)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
how_use_pythonstartup
>>>

It means that we basically were able to hack into interactive mode before starting of it.

This works only on an interactive mode.

Examples of pythonrc

Rasadacrea example of pythonrc gives us possibility to use tab completion :

import readline
import rlcompleter
import atexit
import os
# tab completion
readline.parse_and_bind('tab: complete')
# history file
histfile = os.path.join(os.environ['HOME'], '.pythonhistory')
try:
    readline.read_history_file(histfile)
except IOError:
    pass
atexit.register(readline.write_history_file, histfile)
del os, histfile, readline, rlcompleter

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

How to use python debug with pdb

2018-04-06T18:05:00+02:00

Python has a debugger called pdb. Let's find out how can we use it instead of Pycharm.

To The Point

Calling PDB Stack Trace

There is a PDB class that enables python debugging without additional python packages.

To use it, call stack-trace like that:

import pdb

pdb.set_trace()

The line in which you will put pdb.set_trace() is the one from which pdb will invoke stack-trace.

There is also a way in which you will start function without setting particular line at which stack-trace should start:

import pdb
import yourmodulename
pdb.run('yourmodulename.yourfunction()')

Using PDB - commands to move in pdb stack-trace

PDB stack trace is interactive - in a similar way as in Pycharm.

You can use commands in pdb to analyze stack-trace.

You can either write one letter of the command or full name of command - but some of the commands needs to be written in a full-length.

The most used commands:

u(p) - moves current stack-trace frame up
d(own) - moves current stack-trace frame down
b(reak) [([filename:]lineno | function) [, condition]]
- without argument lists all set-up breaks.
- first argument sets up breakpoint at specific filename/function and line number.
- second argument gives option to break only on specific python condition
cl(ear) [filename:lineno | bpnumber [bpnumber ...]]
- without argument clears list of all set-up breaks.
- with first or second argument, gives option to distinct which breakpoints needs to be cleared.
s(tep) - executes current line
n(ext) - moves to next line of funtion or returns if not found.
r(eturn) - continue execution until current function returns.
q(uit) - quit from debugger.

The full documentation about commands can be found at the pdb documentation page on python docs

Snippets

import pdb; pdb.set_trace()

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

How to Automate web-scraping vocabulary into pdf

2018-04-05T23:00:00+02:00

Lately I've found this vocabulary.com site that give a lot of knowledge. Let's check how can we automate reading of new vocabulary by scraping data from site into pdf.

To The Point

Scraping data from vocabulary.com

First what we need is to gather links from vocabulary.com at Choose Your Words.

Script like that should do that:

import requests
from bs4 import BeautifulSoup

domain_href = 'https://www.vocabulary.com/'
output = requests.get('{}articles/chooseyourwords/'.format(domain_href))
bsobj = BeautifulSoup(output.text, 'html.parser')
distinct_links = {}
for link in bsobj.find_all('a'):
    href = link.get("href")
    if href:
        if "chooseyourwords" in href:
            distinct_links[href] = domain_href + href
print(distinct_links.keys())

After gathering unique links we need to disassemble it.

def disassemble_vocabulary_page(link):
    output = requests.get(link)
    bsobj = BeautifulSoup(output.text, 'html.parser')
    return bsobj.find('div', class_="articlebody")

The script in one piece looks like this:

import requests
from bs4 import BeautifulSoup

def get_vocabulary_links():
    domain_href = 'https://www.vocabulary.com/'
    output = requests.get('{}articles/chooseyourwords/'.format(domain_href))
    bsobj = BeautifulSoup(output.text, 'html.parser')
    distinct_links = {}
    for link in bsobj.find_all('a'):
        href = link.get("href")
        if href:
            if "chooseyourwords" in href:
                distinct_links[href] = domain_href + href
    return distinct_links.values()

def disassemble_vocabulary_page(link):
    output = requests.get(link)
    bsobj = BeautifulSoup(output.text, 'html.parser')
    return bsobj.find('div', class_="articlebody")


if __name__ == "__main__":
    link_to_vocabulary = {}
    links = get_vocabulary_links()
    for link in links:
        link_to_vocabulary[link] = disassemble_vocabulary_page(link)

Saving html data into pdf

Let's use pdfkit to make new pdf. It's using wkhtmltopdf, so first we need to install it with:

sudo apt-get install wkhtmltopdf

Then we can use pipenv install pdfkit to make pdfkit library available at env.

Making use of your html that we have scraped with this script:

def html_to_pdf(link, data):
    pdfkit.from_string(data, "{}.pdf".format(link.split("chooseyourwords")[-1]))

Source code

The output is not yet most prettiest, but still we have data in a pdf.

import requests
import pdfkit
from bs4 import BeautifulSoup

def get_vocabulary_links():
    domain_href = 'https://www.vocabulary.com/'
    output = requests.get('{}articles/chooseyourwords/'.format(domain_href))
    bsobj = BeautifulSoup(output.text, 'html.parser')
    distinct_links = {}
    for link in bsobj.find_all('a'):
        href = link.get("href")
        if href:
            if "chooseyourwords" in href:
                distinct_links[href] = domain_href + href
    return distinct_links.values()

def disassemble_vocabulary_page(link):
    output = requests.get(link)
    bsobj = BeautifulSoup(output.text, 'html.parser')
    return bsobj.find('div', class_="articlebody")

def html_to_pdf(link, data):
    file_name = link.split("chooseyourwords")[-1].replace("/", "")
    pdfkit.from_string(data, "{}.pdf".format(file_name))

if __name__ == "__main__":
    link_to_vocabulary = {}
    links = get_vocabulary_links()
    for link in links:
        html_to_pdf(link, disassemble_vocabulary_page(link).text)

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

How to use sed as a refactor tool replacement

2018-04-04T06:30:00+02:00

Today let's check how to use SED tool that can be used for text replacement.

To The Point

You can replace text in files using sed

sed -i -e 's/few/asd/g' file.ext

We can also make use of ack2 for searching and combine it with sed to create a simplier script than making each call to each individual file.

Let's first find out if ack2 will give as an output list of files in which it will find searched phrase. Using ack2 --help we find out, that -l flag gives us a list of files with seached phrase.

We can now combine those two using only bash script with for-loop.

Let's loop each found file from ack2 and then pass the file path to sed which will replace searched phrase with replacement phrase.

This can then be used as a similar functionality to refactoring name in pycharm (take into consideration it will replace in all files if finds so it's not a full-time replacement for pycharm).

Our script will look like this:

for i in $(ack2 -l "$1"); do
    sed -i -e "s/$1/$2/g" $i
done

To use it, let's call this script refactor_text.sh:

refactor_text.sh some_ugly_name_for_variable some_prettier_variable

Which will look for some_ulgy_name_for_variable in all files from directory you are executing script and replace it with some_prettier_variable in those files which content matches searching phrase.

I've added it into my linux-utils repository as a refactor_text.sh.

Snippets

#! /bin/sh
#
# refactor_text.sh
# Copyright (C) 2018 [Anselmos](github.com/anselmos) <anselmos@users.noreply.github.com>
#
# Distributed under terms of the MIT license.
#

for i in $(ack2 -l "$1"); do
    sed -i -e "s/$1/$2/g" $i
done

refactor_text.sh some_ugly_name_for_variable some_prettier_variable

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Fixing pelican_link_to_title no title problem

2018-04-03T12:15:00+02:00

Yesterday when using pelican_link_to_title plugin I've found a 'no title found' problem. Let's focus on that and fix it.

To The Point

The issue

Yesterday I wanted to use pelican_link_to_title plugin with a file instead of html page. I've forgot that plugin is reading html page which can cause issue while using a file.

Issue I've got while using a link to file was simlar to this:

plugins/pelican_link_to_title/pelican_link_to_title.py", line 36, in read_page
    title = soup.find("title").string
AttributeError: 'NoneType' object has no attribute 'string'

Fixing the problem

Originally the plugin was meant to save page titles. It was meant for html pages. While I used it, one day I added a link to file. It crashed. The problem was rather conceptional than technical - in idea of the plugin.

To fix the problem we need to find, if the link is actually an html page or not.

Checking if link is html page.

While searching for solution for finding if page is html page or not I've found this StackOverflow issue comment that describes a solution of accessing url with HEAD request.

It will give only metadata information - without content - different from request with GET.

The part of solution with head request looks like this in plugin:

r = requests.head(url_page)
if "text/html" in r.headers["content-type"]:
    html = requests.get(url_page).text

Source of plugin fixed.

# -*- coding: utf-8 -*-
""" This is a main script for pelican_link_to_title """
from pelican import signals
from bs4 import BeautifulSoup
import requests


def link_to_title_plugin(generator):
    "Link_to_Title plugin "
    article_ahreftag= {}
    for article in generator.articles:
        soup = BeautifulSoup(article._content, 'html.parser')
        ahref_tag = soup.find_all('ahref')
        if ahref_tag:
            article_ahreftag[article] = (ahref_tag, soup)

    for article, (p_tags, soup) in article_ahreftag.items():
        for tag in p_tags:
            url_page = tag.string
            if url_page:
                if "http://" in url_page or "https://" in url_page:
                    tag.name = "a"
                    tag.string = read_page(url_page)
                    tag.attrs = {"href": url_page}
            else:
                continue
        article._content = str(soup).decode("utf-8")

def read_page(url_page):
    import redis
    redconn = redis.Redis(host='localhost', port=6379, db=0)
    found = redconn.get(url_page)
    if not found:
        header_response = requests.head(url_page)
        if "text/html" in header_response.headers["content-type"]:
            html = requests.get(url_page).text
            soup = BeautifulSoup(html , "html.parser")
            title = soup.find("title").string
            redconn.set(url_page, title)
            return title
        else:
            return get_non_html_page_title(url_page, header_response)
    else:
        return found

def get_non_html_page_title(url_page, header_response):
    file_str = url_page.split("/")[-1]
    file_ext = file_str.split(".")
    url_domain = url_page.split("//")[1].split("/")[0]
    if len(file_ext) > 1:
        # file with extension in url.
        return "Url to {} file: {} on domain: {}".format(file_ext[-1], file_str, url_domain)
    else:
        # no file with extension in url
        return "Url to: {}".format(url_page)

def register():
    """ Registers Plugin """
    signals.article_generator_finalized.connect(link_to_title_plugin)

Snippets

r = requests.head(url_page)
if "text/html" in r.headers["content-type"]:
    # this url_page is an text/html page type content.

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Accessing Polar A300 data with polaraccesslink api

2018-04-02T17:10:00+02:00

Lately as a gift I got a Polar A300 watch that gives a lot of data to user. There is a web and mobile application to it. Check out the API that you can use to access data for your own application!

To The Point

API

There is PolarAccessLink API from which you can access data gathered by A300 watch.

From this API I've found python example usage for polaraccesslink on github.

To create your ClientId and ClientSecret used in the polar-example, first create your Polar account and then login at PolarAccesLink Admin.

Then click on Create a Client, fill all the forms and as a result you will have ClientId and Client Secret.

Reusing Example code

First let's clone our example repository with:

git clone https://github.com/polarofficial/accesslink-example-python.git
cd accesslink-example-python/
pipenv --python 3.6
pipenv install requests

Update the config.yaml with ClientId and ClientSecret.

Then let's create OAuth access token, first starting auth_service available at localhost:5000:

pipenv run python authorization.py

Access the localhost:5000 from which you will login to your own UserId and create your OAuth Token for application.

You can now use the api with example and extend it's usage beyond what is available at :

pipenv run python accesslink_example.py

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Ascii Easter bunny from picture

2018-04-01T12:32:00+02:00

Happy Easter!

Let's find a script that enables us to change image to ascii text!

To The Point

Img2Txt

Firstly let's check img2txt.

Installation

To install that type of utility I use pipsi. Install using:

pipsi install img2txt.py

Testing img2txt

Since it's Easter I decided to check some of the Easter-Bunny pictures as in this blog to use it with img2txt.

Image I'm going to use to check img2txt: .

And the result looks like this:

---------;;-;------:::::::-:::::;;;;;;.;;;;;;;--:!>>7777777777777777777>7777777777777777777777777777
-------;;;;;;------------------;;......;;;;;;;;;-:::!!!!!!!!!!>>>>7>>>!::!>>777777777777777777777777
-----;;;;;;;------------------;..................;;;;:!:::::::!!!!:-;;;...;;;:7777777777777777777777
-;;;;;;;;;;;------------------;......................;-:::::::::::;..........;77777777>>>77777777777
;;;;;;;;;;;-----------;;-----;;......................;;::::::::--;..........;;-7777>>>>>>>>>77777777
;;;;;;;;;;----------;;----;--;;.............;.......;;---------;..............;>>>>>>!!!>>>>>7777777
;;;;;;;;;;;;------;;;-;;;;;--;;;..........;;..........;;-------;...............-!!!!!!!!!>>>>>>7777>
;;;;;;;;;;;;;;-;;;;;;;;;;;;;-;..;;;;;;;;;;;;;.............;;--;................;-::::::::!!!>>>>>>>>
;;;;;;;;;;;;;;-;;;;-;;->>!;7Q?!;.!!.!..;;;;;;..............;;;;................;------::::::!!!!!>>>
;;;;;;;;;;;;;;;;;;;;;.....:$?!>$C;>7!:;!.;;;;..............;;;;...............;;;-;;----::::::::::::
;;;;;;;;;;;;;;;;-;..;:!7CO$$C!7$C$O;OC>!7-;;;;;;...........;;;;...............;;....;;----:::::::---
;;;;;;;;;;;;;;;;;7:-;!7COOOO$OC$$OO?O!$O;;-.;;;;............;;;......................;--------------
;;;;;;;;;;;;;;;!.>?>-$$O$Q$O$$O$Q?!?!O$7C7C;:;;;;;;.........;;;;;...................;;--------------
;;;;;;;;;;;;;;.>?-;7$$$$O$C!C?!>O77!?$$>>CC!;;;;;;;;;;;...;;;;;;;;;...............;;;;;;----;;;;;;;;
;;;;;;;;;;;;;-O.:O$OOQC>$$?C$7:$!7>>OQ!>7?7!7C;;;;;;;;;;;;;;;;;;;;;;;...........;;;;;;;-;;;;;;;;;;;;
;;;;;;;;;;;;.-7?O7$$?COO-C>7$C!>?7!:C-7>7!7>7-!-;;;;;;;;;;;;;;;;;;;;;;;.;;;;;;;.;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;!?>C;O$$$CO$;OQ?7$>->??!!?-777!CO>?$!-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;:C--:7$$$.7:O$;CC;Q:CO7!?!>-77>!O$O7-!-;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;C;.?$7OO>OOO;QCQQCOC$C!C!>7:7?7!?$$$7O!:;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;!-.??:O?!C:$OO-$COCC?>::!-C?->7!7:QCOO:>!;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;.;;;;;
;;;;;;;;.7:7>;$$C;CCOQ$OQ7O-!...--:!?:>7!7!OO>$7!--;;;;;;;;;;;;;;;;;;;;.;;;;;;;;;;;;;;;;;;;;;..;;;;;
;;;;;;;.-!>C;OOC$O?.C$C>CQ-  .:!!:->O>!>O7C>CO$??!:;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;....;;..
...;;;;.--7:!O>-C$C?:CO>:: .;:!?!::!!>::$77Q$$QO??!:;;;;;;;;;;;;;....;;;;;;;;;;;;;;;;;;;;;;......;..
........:7C-$$$CO>$OO!CO;..:???7?77!C?>?>CQ:$-$O>?>>;;;;;;;;;;;.....;;;;;;;;;;;;;;;;;;;;;;..........
........:?.7O?OO>COQC7:?..:>77?--::7??7O!CQO$7?$?!7>-..;;;;;;;.....;;;;;;;;;;;;;;;;;;;;;;;..........
.......-7:?>OCOOQO-?Q?:-..!??C!>?!:>?>C$!!?7CO$$C>>>:....;;;;.....;;;;;;;;;;;;;;;;;;;;;;;;..........
.......-C:??!O7:C$?:OOO-..?OC>$HH??QC>>>--:O$$O$O?!77;............;;;;;;;;;;;;;;;.;;;;;;;;..........
;.....;:;7:$O?C7;7O$OC:..;?C7-$HHO>C7!;:::!!CQ$OO?>>C;............;;;;..;..;;...;....;;;;;..........
;;;;;..>>!>QO$OC7;Q$$?-..->C!:O$HO!!:!!::-!::O$COC7-$;............;;;;..................;;..........
;;;;;..:>:!$$;-!OQ$?77;..!77?!>O>!?!>:-!!;!!-!$$OC:-C;................................;;......;;;.;.
;;;;...:!7>OO!!7CO7::!...-:!:-!7?!------!-7:-:?CO?:7!;...............................;......;;;;;.;.
;;.....>!-7$OC$Q$7--:;...;77:7!>!!:---!!!!>!::>?O7!!-;.....................................;;;;;....
;......>!;:C7C$O?>:!!;;-;.!C?-77!!-::>>!:7>-!-!>?7>!-...............................................
;;;;;;.>!;-C-O$??>:7!-.!;;-CO?!?C>!7!>!:!>-!-!->C7>-?..;.;;;;;;;....................................
;-;;;;;!!;;?!O$C>7>7>:;:.;->7?C7>>>77:!!!-:!!!:?C!!!:..;.;;;;;;;;;;;;;;;;.;;;-;;;;;.................
-:-;::;:>-.??$O7>7??!!:-;;;->!OC>?7>!---;:>!:-->C>::;;;;;.;;;;;;;;;;;;;;;;--:!:>>!:--;;-;;;;;.....;;
7CC?CC>?>-::$!O?>7!>7>>!--;;::>7?>:--;-!!!:>>::>?:O7-::>:->:--!>!!::!!>!!7>77?????C77>77>>!!!!::::!!
OCO?>;-?:--7O$CO>!>>>>!>::7-:->!!:--;;-:C>7:!:-7>>-;?7?7????77777?????77?????CCCCC????C???C??7?7??77
?7;;. ;--::;$O>;   ;>>>!:;..;-!!:--;;;;!::!!;.   .>!77?C?7777777?C??????CCC??C??CC?CC?CCC??C???7?C77
7------:::7-;;.      :?!:>>;-. ;--;;;;;-:7!...!!!!;;7>777>! ; !.77????7??????C????C77?77777>>>>>7777
-  -::-. !!   .7 ? ;  :;>>>;- : ;;-;;;;;-!..>>!----!:7>77 !;  !. !????77?>7?7777?7>?7?77>>7>>>777>>7
   :!!:. ::   .: !:  !>77777.-;-; --;;---.>>:-;;;..;-:?>  ->!!>>>!!7?7C7??777777??7???777>C7>77>>777
   ..!-!!C    ;  ?  ;77777777.-- 7!----;-7:-.    .-;. 7!:>7;!>!>!:--7>>77C77C??CC?>CCC?C?7777?7>7??7
;    :.;!     . ?   > .777. 777777. .::-7-;  .:!!!!!!!;?7>-::?77??!!::?77?>7C7>777!?777777?7>>7777??
!.    ;;   >       >.:;777.;:7777?7  :!:-;.;!!!!!!!!!!!?7?C777??77?77!!777>C7C??7>C7:?7>>7>?77??COC?
    : - :         .>;.7777777?????: !!!;..!!!!!!!!!!!!!:C?C??????!!;?7>>>C777>7>7777>7777?>C??!7CQC7
      :    ..  > .777777777???????- >.>7;!!!!!!!!!!!!!!!CCC??????7:-7?>;>:77OC77?7?>?7!77>??CC7?>?77
 ?    .  .: ! .  .7???????????>???; 7.!!!!!!!!!!!!!::!!!OOC???????????7777?OCC?C!??7?>?7COOC7?7?>7!7
 C    .  ! ..>.. ;?!;????????.:.?C7 ?.;:!!!!!::-;;;;;-;-:OOOC????????7777?C?7>$>C?7>>>!7:OO??>OCO:>:
.....;;  ..-....;>?!?????????.!>CC!.C;>:!!!!:--;;;;;-----CCO>CCCCC???????O$CC7O>O7:7>!CO!7>7-O>$!7??
....O-....!.>...;????- !..???->:CO?;O-7!>>!:--;!C???CC---$$>>7OOOCC???C?$C?7>>O7>>?>7?!7?7-7?CCO?7C?
...;;>.>.;.7...;;>CC. .. .?CCC>7OO$:$7;7>>!:;C??C????CCO-Q$>Q?O$OOOOOOCCOO:O>?7!7:C?:?O-O7:C7?>C>??O
.;;-!Q;....;..;-:!C:.777>..:CCCOO$OQQ--H7>!CC???C?C????CQHQQ$$$Q$$Q$O7$$77>!7?CC7>!C?7?>?>!!?7>?>7?>
;:-!OQ;:;..;-;;H!!C!7>;-;>.;OOOO$$HH$$?HO>$C???!-::-:7??NNQQHQQ$QQQ$OQ>7C?7?O?$$C:777OOCO:!:>>C?777C
-7!?QH:;;:?;;--NH7O;?!??7.7;O$$$OQNHH$H$HCOC?:---;-:::!>N$NQHQHHHO$QQQC$$C>!-?O>:?7C7!??O;??!:O?>>?>
!7OHHQQ:->?::!:>7C$;!?C>C>C;-$Q$OC??ONQ$$QQ7?7!!-;----!$H$NQNNNNQQQQ$7?CO!7:O$?O77??OO:?C!?>?>?7O??C
ON$$??CO>>C>77Q7CO$>--:-7??:$QQHHQHQOQ?QHHHC?O!:::::!>?HHQOOCOC?7Q$H$COC77O?:?O:CCCC?>7C7>?O:>C7>C?>
QQOO?$Q7CC?$C$C?$>$O>Q?O>!->OHHNQ$$$HO$$QQHHQQ?7>777?C$QO$>?$$?$HOOCCO77-OOQ>>::?COCC?O??!?777:-77>>
$$?O>$?>O?!CC$77????O?>C!OCC>ONHCQOQQ$Q$OOOQ$$$H?C7?OCQ$O??COCO$7OOO!?C>!7C7C>7???>>??CO??O77C!!!>C>
OCQ?QO7CO?O?CO7?C7?CC7C$C7O$HQ7O$$$$CQ77??$>CQ$COCCCOC?C7??OOOO>7C$C77>C?>>>CC7C7>OC?>$$CC?7!:>7777:
?OO$7O$?OOOCO?C?>?!7?C77C?7HOO?QOCOOC7O?7OOQ$CQOO$CCOOOC7CO$??!>77?7O7!OC7?C?>777:CC?77C?$7O>:!7!C>!
$C?!>QQCQOCOOCO77:?7$7?O?>CH$?7OC>!CO$CC>CC??OC?COOOOCC>OO?CO77C!77!CC?C$7?>??!>:!?$?7CCCQ?C7>!77>::
7>77QO7CQOCOC77?>777?CO7COOO>77?7?>CC?CCCOC77OCO>7O$?C7>!OCC!7C>??C7>O7?>C::C7>?>!77??OCC$COO77?:>:!
7C?$$C?O?77>C??C7>QCCO7?7?C?7?OC??CCO$$7OCC???O>?!$7>CC7?7?7??C?OC??O?C?77?>7?7?>:!!O??O$$COO?>>7!77
?CC$?C???::7:?CC?Q>7C?C>7C7:7CCOOCCOO$CCO!?OO$CC>?>>77??7??OOCCCOC$?!?O7C?>??OC?77:>CC?O?CCOO?>7>!77
$OOCO?C?7>!7?CC7C>>?!??OC?7??OO$OQ$OQO>7?!C7$???C??7??7777>?COOCOCC7C??CC??7C$C??7!CCCC??>7CO?>>?7??

IMHO it does not look great.

Let's give it a second shot with this image:

The output:


                                                                         :           

                                                                        M            
                                                                       M >           
                                                                      M  C           
                                                                    !    Q           
                                                                   M     H           
                                                                 :C      !           
                                                                M                    
                                                              -H                     
                                                             M     M  M              
                                                            M    M                   
                                                           ?    !                    
                                                         ;    M        -             
                                                             M           !           
                                                        :   ?            M           
                                  MMMMC                M   .          M  -           
                              M?        M             M  Q           !  M            
                            M.           O                              -            
                          M         M-CM.O           M  M              M             
                         M         M M                 M                             
                       .;             :             M  ;              M              
                       ;          M   M             - :              M               
                      >               M  7            M             Q                
                     H           M    N  M         M               H                 
                    M                    !         M                                 
                                M    >  >          Q M           H                   
                   M            ;    M  M            M          Q                    
                               M     M  .            ?         M                     
                  M            $     ;               -        M                      
                  M           ; ;      Q                     H                       
                  7           M M      M             -     O                         
                  ;                  ; M  MM> M      ;    M                          
                  :          M   M   M M   ? M MM        7                           
                  M         M     M    M M-     M  H   :CC.                          
                  M       M        M    MQ             M   ;M                        
                         H        M>                          $                      
                   M    M       M                              $                     
                    M  Q      M                 O                                    
                     . H     M              MM MM               M                    
                      M     M       M       MM  M-                                   
                           !         M      -MMM        M        Q                   
             C?;                   MM  N     MMMH       :$      MM                   
         M77    ?MMM      M           ;O C    MM    ;CM         MM                   
      .M   M. M:MMM. M7   QM             $ M  !   M7   QMCCM     M                   
     C  ?MM?.  M    M  M   !     >.    CM  .M O  M  H  ;M        .                   
    M    M  M?. M .M   MM;Q         .       - C M:    -           ;                  
       MMHM M  >M M  >MMM MM        M Q       N                 M    ;-              
   M   Q.    M   M  Q:  M MO        M   -     M      ?          HM  M                
        M MMMHM M   M   !M                    M     M           OH     -             
  M   MMMM M   M   MMQ M.  M              .  :     M               O      ?          
  M   ;H    M  M  M .MN                    7 NMMMM!               M       M          
  M    M M$MMMM  M   N?  ;  M               M   M              M :                   
  M  MMM.:N  O  C!   M    : Q              M    M              M         $           
  M  H    M  M  MH  M    HM7>7          ->M     MQ            M           N          
  M   M MMMMQ  M MM?   >M  : O        M;          M                       ?          
    ?MMM!   M  H   M    M :M  M      M             M         M            M          
    M    M N  M   M     M O    M                            M           !M           
   M$ MMMMMM  7   ?   M:  O    MM   ;                     ;!      MM;                
   M M M   ; M-M M    M MM.    $  :M               M    -M      M                    
    CM >  M  M   M    M M         M              MMMMMO      .N                     .
    M   M M  -  M   MN  MM MM MM M               >    H     M                        
     MMMMMM H   M   M MM  M  N  M                    MM    H                         
      M   O M?M C   - MM        N                     !   MM  M;                     
      OM  - M M    MM>          M                     M   Q     ;     HM             
       MM 7 M  ?   M C         MM        M>CM7  :$    M  M      !    N !            .
        H M M   MM  $>       C          MQ  H $M                 M   ;M              
         OM M OM MMMM.M     O    O      M    M  :   O>           >  M .              
           M 7MQ  MM M           !     !         M   M            ;  M M             
            $M    - :H     M      M   MM         >M               MM MMM   M:        
              NQM  MMM              M M          ;MM              MC MM M?  MN       
                MMMMNCM   :                                       M   MMM   MM.      
                 MM> MM   Q                         M             M   MM    M .      
             >N.NM N   MM ?        ;                M             M    M   M  Q ;MM  
               H7C M  O   M                         M             M     ;$.          
                           7                        :             M                  
                           M     M                                M                  
                                                   M              M                  
                            M   7                 MM           -M M                . 
     .                       M  M    M !          ;.       H>    MM                  
                                M   M   M;       M        M      ;                  .
                              ! !   M            CM M    >      O                    
       .                       M    !           M  M M   ;        ?                  
                              MMN    M         H    H      MNNM   M >M               
 .                             ;M>    ?   M          :           : M$                
             .               ;$MMM     .M.    !                  . :MO               
                          MMMMMM  M          .         .:       M    NMM:M;          
                         :         M       M             M    :$                     
                                     M7:$M                MMMM                       
                                                                              .      

                                            OM.                                      
                                          7M !                                       
                                      CMMMM M                                        
                                       ; MM M 7.MMM.                                 
                              .                                            .

MotionDesignStudio script

There is also another ascii movie image ver 1 py.

The Readme is very convenient to know how to use it.

Fixing issues with script

Well... I tried to use the script itself, but unfortunatelly after a bit of digging I've found it lacks proper information about package used and has some issues.

First of all I had few TabsError (inconsistent use of tabs and spaces in identation).

So I went to vim and used vim command :retab. Saved file.

Then I needed to fix dependencies problem.

So I figure out why not create project out of it to have all the relevant files in one place.

As the result here is the link to project at github. It contains Pipfile and Pipfile.lock - for easier usage with pipenv.

Usage

I suggest you to get the version from my github project since it has fixed some issues.

git clone https://github.com/anselmos/ascii_movie_image_ver1.git

Then :

cd ascii_movie_image_ver1 && pipenv install --python 3.6
pipenv shell

Using on pipenv shell:

python ascii_movie_image_ver_1.py image.img 4 16

Results of using:

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Debugging https problem on pelican_link_to_title plugin

2018-03-31T22:20:00+02:00

Yesterday I've added into my usual <ahref> </ahref> html tag https link that failed. I was surprised, but since it was not my main focus then I decided to give it a shot today. Let's check out what is wrong with plugin or what is wrong with the page I used.

To The Point

The failed https page link

The link to the page was actually the Travis-Ci page: <ahref>https://travis-ci.org/beyondgrep/ack2</ahref>

It will fail with this:

plugins/pelican_link_to_title/pelican_link_to_title.py", line 34, in read_page
    r = urllib.urlopen(url_page).read()
File "/usr/lib/python2.7/urllib.py", line 87, in urlopen
    return opener.open(url)
File "/usr/lib/python2.7/urllib.py", line 208, in open
    return getattr(self, name)(url)
File "/usr/lib/python2.7/urllib.py", line 437, in open_https
    h.endheaders(data)
File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders
    self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 864, in _send_output
    self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.connect()
File "/usr/lib/python2.7/httplib.py", line 1220, in connect
    self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)
File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
    ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 243, in __init__
    self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 405, in do_handshake
    self._sslobj.do_handshake()
CRITICAL: IOError: [Errno socket error] [Errno 1] _ssl.c:510: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error

The first thing I've found that may be relevant is to change urllib library into requests - as it is shown in this issue of pytorch.

Let's check out if it will help.

Replacing urllib into requests.

After replacing urllib with requests (adding import and changing gathering data from page) pelican_link_to_title looks like this now:

# -*- coding: utf-8 -*-
""" This is a main script for pelican_link_to_title """
from pelican import signals
from bs4 import BeautifulSoup
import requests


def link_to_title_plugin(generator):
    "Link_to_Title plugin "
    article_ahreftag= {}
    for article in generator.articles:
        soup = BeautifulSoup(article._content, 'html.parser')
        ahref_tag = soup.find_all('ahref')
        if ahref_tag:
            article_ahreftag[article] = (ahref_tag, soup)

    for article, (p_tags, soup) in article_ahreftag.items():
        for tag in p_tags:
            url_page = tag.string
            if url_page:
                if "http://" in url_page or "https://" in url_page:
                    tag.name = "a"
                    tag.string = read_page(url_page)
                    tag.attrs = {"href": url_page}
            else:
                continue
        article._content = str(soup).decode("utf-8")

def read_page(url_page):
    import redis
    redconn = redis.Redis(host='localhost', port=6379, db=0)
    found = redconn.get(url_page)
    if not found:
        r = requests.get(url_page).text
        soup = BeautifulSoup(r , "html.parser")
        title = soup.find("title").string
        redconn.set(url_page, title)
        return title
    else:
        return found

def register():
    """ Registers Plugin """
    signals.article_generator_finalized.connect(link_to_title_plugin)

After a bit of tweaking my pipenv environment I got it finally working.

Firstly I installed only requests with pipenv install requests, but I had an issue :

local/lib/python2.7/site-packages/urllib3/util/ssl_.py:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings                  
    SNIMissingWarning


local/lib/python2.7/site-packages/urllib3/util/ssl_.py:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    InsecurePlatformWarning

CRITICAL: SSLError: HTTPSConnectionPool(host='travis-ci.org', port=443): Max retries exceeded with url: /beyondgrep/ack2 (Caused by SSLError(SSLError(1, '_ssl.c:510: error:14077438:SS$ routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error'),))

I've found that installing the security addon will fix the above problem - pipenv install requests[security].

Check out latest master of the pelican_link_to_title plugin that contains this fix and redis-optimization.

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

How to use ack2 searching tool for code

2018-03-30T23:55:00+02:00

Ack2 is a greplike tool. Accordingly to documentation it's primary purpose is for searching large amount of content data like source-code of projects.

There is a great comparison chart on ack2 authors site that I recommend to check!

To The Point

Docker for building ack2

So as I'm a docker-fan-boy as usually I decided to check if this tool will work, but not on my host machine, but using docker first with a ubuntu:trusty docker-image.

I've created a Dockerfile like that:

FROM alpine:3.6
RUN apk add --update
RUN apk add --update perl && rm -rf /var/cache/apk/*
RUN apk add --update wget && rm -rf /var/cache/apk/*
RUN apk add --update tar && rm -rf /var/cache/apk/*
RUN apk add --update curl && rm -rf /var/cache/apk/*
RUN apk add --update make && rm -rf /var/cache/apk/*
WORKDIR /app
RUN curl -o ack2.tar.gz https://codeload.github.com/beyondgrep/ack2/tar.gz/2.23_02
RUN tar -zxvf ack2.tar.gz
WORKDIR /app/ack2-2.23_02
RUN perl Makefile.PL
RUN make
RUN PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install File::Next'
RUN make test
RUN make ack-standalone
RUN cp ack-standalone /bin/ack2
CMD ["ack2"]

Run with docker build -t anselmos/ack2:2.23_02 -f Docker.alpine.latest . and checked for building to be done.

Ack on Docker alpine

There is an alpine version of ack but only on edge version and the ack version is 2.18. I used this Dockerfile to check it:

FROM alpine:edge
RUN apk update
RUN apk add ack

With docker build -t anselmos/ack2 -f Dockerfile_alpine . And then docker run -it anselmos/ack2:alpine

Installation

There is also a much more simpler way that is described here.

For ubuntu you run:

sudo apt-get install -y ack-grep

And you can create a soft link for calling ack not ack-grep with:

sudo dpkg-divert --local --divert /usr/bin/ack --rename --add /usr/bin/ack-grep

For Ubuntu:Trusty it's 2.12 version.

Usage

To use ack check this documentation - contains all the reference - also I encourage you to check ack2 --help to see what type of flags there is.

The output from ack2 is a bit different from grep, and it's more readable than on grep.

ack2 "perl" *.md

And it will search for "perl" in all .md extension files.

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Regexp vs clean python code performance check

2018-03-29T00:30:00+02:00

There are re library that gives opportunity to use regexp. With them you can make some of the string-checks that using clean-pythonic way would be a bit more complex.

Let's check out if this is really a kind of bottleneck that can underperform in some edge-cases!

To The Point

Regexp example.

Let's say that we have a string that we know should contain max 16 regular text letters.

With Regexp we could check if the string matches our expectations using this code:

import re
string1 = "Test"
regexp = '^[a-zA-Z0-9]{16}$'
pattern = re.compile(regexp)
print(pattern.match(string1))

The printing will return "None" - meaning that variable string1 is not matching string1.

Performance regexp

Let's now check how to find performance of it.

First we need to change our script into function:

import re
string1 = "Test"

def check_regexp(string_variable):
    regexp = '^[a-zA-Z0-9]{16}$'
    pattern = re.compile(regexp)
    return pattern.match(string_variable)

print(check_regexp(string1))

Now the timeit library comes into play.

time_perform = timeit.timeit("check_regexp(''.join(str(n) for n in range(8)))", setup="from __main__ import check_regexp", number=100000)

It will check timing of the function.

Let's check performance between the clean-pythonic way and the regexp way:

import re
import timeit
import string
string1 = "Test"
regexp = '^[a-zA-Z0-9]{16}$'

def check_regexp(string_variable):
    pattern = re.compile(regexp)
    return pattern.match(string_variable)

def check_python(string_variable):
    if not len(str(string_variable)) == 16:
        return False
    alphabet = string.ascii_letters + string.digits
    for char in string_variable:
        if char not in alphabet:
            return False
    return True

time_perform = timeit.timeit("check_regexp(''.join(str(n) for n in range(8)))", setup="from __main__ import check_regexp", number=100000)
time_perform2 = timeit.timeit("check_python(''.join(str(n) for n in range(8)))", setup="from __main__ import check_python", number=100000)
print(8 , time_perform, time_perform2)
time_perform = timeit.timeit("check_regexp(''.join(str(n) for n in range(16)))", setup="from __main__ import check_regexp", number=100000)
time_perform2 = timeit.timeit("check_python(''.join(str(n) for n in range(16)))", setup="from __main__ import check_python", number=100000)
print(16 , time_perform, time_perform2)
time_perform = timeit.timeit("check_regexp(''.join(str(n) for n in range(32)))", setup="from __main__ import check_regexp", number=100000)
time_perform2 = timeit.timeit("check_python(''.join(str(n) for n in range(32)))", setup="from __main__ import check_python", number=100000)
print(32 , time_perform, time_perform2)
time_perform = timeit.timeit("check_regexp(''.join(str(n) for n in range(64)))", setup="from __main__ import check_regexp", number=100000)
time_perform2 = timeit.timeit("check_python(''.join(str(n) for n in range(64)))", setup="from __main__ import check_python", number=100000)
print(64 , time_perform, time_perform2)

And we see that the results are outperforming regexp with clean-pythonic function. It's not like there's huge difference, but check yourself on this results:

8 0.5771335330209695 0.38084229797823355
16 0.818426341022132 0.6164886929909699
32 1.226193142007105 1.0479984590201639
64 2.1883155060058925 1.9493357640167233

It was only on some digits. Let's check now how it will perform on string with digits:

import re
import timeit
import string
string1 = "YetAnotherStr123" # has 16 chars
string2 = "YetAnotherStr123"*2 # has 32 chars
string3 = "YetAnotherStr123"*4 # has 64 chars
string4 = "YetAnoth*rStr123" # has 15 chars and one special
string5 = "YetAnotherStr12*" # has 15 chars and one special
regexp = '^[a-zA-Z0-9]{16}$'

def check_regexp(string_variable):
    pattern = re.compile(regexp)
    return pattern.match(string_variable)

def check_python(string_variable):
    if not len(str(string_variable)) == 16:
        return False
    alphabet = string.ascii_letters + string.digits
    for char in string_variable:
        if char not in alphabet:
            return False
    return True

time_perform = timeit.timeit("check_regexp(string1)", setup="from __main__ import check_regexp, string1", number=100000)
time_perform2 = timeit.timeit("check_python(string1)", setup="from __main__ import check_python, string1", number=100000)
print(time_perform, time_perform2)
time_perform = timeit.timeit("check_regexp(string2)", setup="from __main__ import check_regexp, string2", number=100000)
time_perform2 = timeit.timeit("check_python(string2)", setup="from __main__ import check_python, string2", number=100000)
print(time_perform, time_perform2)
time_perform = timeit.timeit("check_regexp(string3)", setup="from __main__ import check_regexp, string3", number=100000)
time_perform2 = timeit.timeit("check_python(string3)", setup="from __main__ import check_python, string3", number=100000)
print(time_perform, time_perform2)
time_perform = timeit.timeit("check_regexp(string4)", setup="from __main__ import check_regexp, string4", number=100000)
time_perform2 = timeit.timeit("check_python(string4)", setup="from __main__ import check_python, string4", number=100000)
print(time_perform, time_perform2)
time_perform = timeit.timeit("check_regexp(string5)", setup="from __main__ import check_regexp, string5", number=100000)
time_perform2 = timeit.timeit("check_python(string5)", setup="from __main__ import check_python, string5", number=100000)
print(time_perform, time_perform2)

Well... now we can see what type of problems our clean-pythonic code can have with the results as follows:

0.1520436129940208 0.14696863299468532
0.12985941700753756 0.0332289119833149
0.13141735998215154 0.03479363300721161
0.1242920590157155 0.1090795070049353
0.12702590200933628 0.14172734299791045

It looks like in the string5 version pythonic code underperforms.

For now I don't see any valid solution that could outperform this problem.

I will come back to this if I find any valid solution.

Acknowledgements

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Python typing support for type hints - type aliases and NewType

2018-03-28T21:40:00+02:00

Type Hints/Typing were introduced at #PEP-484 and described in more detail at #PEP-483. They look very familiar to Java Collections. Nevertheless they are a types that can be extended at your own purpose.

Documentation about typings is really straight-forward. I recommend you to check it out.

To The Point

NewType

You have types in python. Within them you can create a new types by making an inheritance.

I.e. to make new type from int you would normally do:

class NewIntType(int):
    pass

But using typing you can do this:

from typing import NewType
NewIntType = NewType('NewIntType', int)

class NewIntInheritance(int):
    pass

def sumUp(a: NewIntInheritance, b: NewIntInheritance) -> NewIntInheritance:
    return a + b

print(sumUp(NewIntInheritance(1), NewIntInheritance(2)))

from typing import NewType

NewIntTyping = NewType('NewIntTyping', int)

def sumUpTyping(a: NewIntTyping, b: NewIntTyping) -> NewIntTyping:
    return a + b

print(sumUpTyping(NewIntTyping(1), NewIntTyping(2)))

They will basically do the same. There is a difference. When you create a new type with inheritance, that you can use isinstance with it.

This code shows the difference:

NewIntTyping = NewType('NewIntTyping', int)
def checkDifferenceInheritance(a: NewIntInheritance):
    print(isinstance(a, NewIntInheritance))

def checkDifferenceTyping(a: NewIntTyping):
    print(isinstance(a, NewIntTyping))

checkDifferenceInheritance(NewIntInheritance(1))
checkDifferenceTyping(NewIntTyping(1))

It will output with this:

True
Traceback (most recent call last):
    File "test1.py", line 38, in <module>
        checkDifferenceTyping(NewIntTyping(1))
    File "test1.py", line 35, in checkDifferenceTyping
        print(isinstance(a, NewIntTyping))
TypeError: isinstance() arg 2 must be a type or tuple of types

So using typing - you should avoid isinstance or even issubclass - as mentioned in TypeVar section

Type Aliasing

There is also aliasing of types within typing. You can create aliases from other types as documentation at type aliases says:

from typing import List

ListOfFloats = List[float]
print ListOfFloats([1.0, 2.0])

Where is this usefull ? Let's say you want to annotate your function to return List of numeric type of data.

You can do that with:

from typing import TypeVar
from typing import List
T = TypeVar('T', int, float)

def sort_list(elements) -> List[T]:
    return sorted(elements)

print([1,2,20, 5, 4, 8, 3])
print(sort_list([1,2,20, 5, 4, 8, 3]))

Adding to this decorator that we have introduced at this article :

def decorator_annotation_parameters(function_wrapped):
    def wrapper(*args, **kwargs):
        annotations = function_wrapped.__annotations__
        output = function_wrapped(*args, **kwargs)
        returned_type = annotations.get('return')
        for kwarg in kwargs:
            kwarg_type = annotations.get(kwarg)
            assert type(kwargs[kwarg]) == kwarg_type
        if returned_type:
            assert type(output) == returned_type

        return output
    return wrapper

We can create a checker that will check if the output value consists of Float-Lists and matches annotation output type.

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Debugging tmuxp focus problem

2018-03-27T22:35:00+02:00

To The Point

The `focus` problem.

So there is an issue #326 of tmuxp repository that I also found using tmuxp. It's about using focus attribute that does not work.

Debugging

So first of all I have started installed tmuxp with pipsi with:

pipsi install tmuxp

Then I have found that pipsi actually creates virtualenv to run scripts.

So I've found where actually tmuxp is located with:

which tmuxp

That outputed with:

~/.local/bin/tmuxp

And found location of the script that should be similar to:

~/.local/venvs/tmuxp/lib/python2.7/site-packages/tmuxp

So first I tried to debug how the Tmuxp works internally and looked for focus in code.

I've finally found the tmuxp load uses code with this stack trace :

at cli.py command_load invokes:
at cli.py load_workspace then invokes builder class:
at cli.py builder.build from workspacebuilder: - at workspacebuilder.py
and in cli.py after builder.build() - invoking set_layout_hook(builder.session, 'client-attached')

After tampering and making printing and tracing down the problem I've found that making sys.exit() after builder.build() creates tmux session that properly works with focusing.

So after that I narrowed down problem and found that commenting set_layout_hook at cli.py fixes problem.

I have created a #371 PR and requested merging to branch. Let's check out what the maintainer has to say about my fix which frankly is more of a hack.

Acknowledgements

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Command_T for fuzzy search in vim

2018-03-26T19:50:00+02:00

Command T is a Vim Plugin that is Ruby based. It enables to make a fuzzy search for files and in files.

To The Point

Installation

To install Command_T you need to have ruby addons in vim.

Check out if you have ruby-support in vim with vim command:

:ruby 1

If you are so lucky as I am and you have following output:

E319: Sorry, the command is not available in this version

Than you have to add ruby-support to your vim or install vim with pre-existing ruby support.

Installing Vim with Ruby support

If you use Ubuntu system, you can install vim with ruby support with:

sudo apt-get install -y vim-nox;

For other, I'd suggest to build vim with commands shown in this stackoverflow answer:

git clone https://github.com/vim/vim vim
cd vim
./configure --enable-rubyinterp
make
sudo make install

Installing vim-plugin.

After enabling ruby support in vim, you need to add this to your vimrc using Vundle :

Plugin 'wincent/command-t'

Usage

To use you can invoke it with <Leader>t - so the \t standard vim combination.

Well ... My first usage outputed with this beautiful error:

command-t.vim could not load the C extension.
Please see INSTALLATION and TROUBLE-SHOOTING in the help.
Vim Ruby version: 1.9.3-p484
Expected version: [unknown]-p[unknown]
For more information type:    :help command-t

What it means? it means that we need C extension for the plugin to work properly.

The documentation with this installation guide gives information that we should go to vim-plugin folder:

cd ~/.vim/bundle/command-t/ruby/command-t/ext/command-t

And compile extension with:

ruby extconf.rb

But my system does not have all ruby-extensions and building tools, so it failed with:

Unable to require "mkmf"; you may need to install Ruby development tools
(depending on your system, a "ruby-dev"/"ruby-devel" package or similar).
[exiting]

Let's install them:

sudo apt-get install -y ruby-dev

And now checking again compilation goes with this effect:

ruby extconf.rb 
checking for float.h... yes
checking for ruby.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for fcntl.h... yes
checking for stdint.h... yes
checking for sys/errno.h... yes
checking for sys/socket.h... yes
checking for ruby/st.h... yes
checking for st.h... yes
checking for pthread_create() in -lpthread... yes
creating Makefile

Now let's make this.:

make

Outputs with:

compiling heap.c
compiling ext.c
compiling matcher.c
compiling match.c
compiling watchman.c
linking shared-object ext.so

So it means we are good to go ?

Checking... Yeah!! It works.

Snippets

Installation

sudo apt-get install -y vim-nox;
sudo apt-get install -y ruby-dev;

VimRC:

Plugin 'wincent/command-t'

Building plugin:

cd ~/.vim/bundle/command-t/ruby/command-t/ext/command-t
ruby extconf.rb 
make

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

How to use NERDTree

2018-03-25T21:30:00+02:00

The NerdTree is a vim plugin that enables us to explore files within vim.

Installation

To install NERDTree you can use Vundle with adding this to your .vimrc:

Plugin 'scrooloose/nerdtree'

Usage

All you need to do is either use vim command :NERDTree or map vim-command with Ctrl+n (or other combination):

nmap <C-n> :NERDTreeToggle<CR>

To learn some of the NERDTree key-combinations check the "?" sign at NERDTree, here is what you will find:

" NERDTree (5.0.0) quickhelp~
" ============================
" File node mappings~
" double-click,
" <CR>,
" o: open in current window
" t: open in new tab
" T: open in new tab silently
" middle-click,
" i: open split
" gi: preview split
" s: open vsplit
" gs: preview vsplit
"
" ----------------------------
" Directory node mappings~
" double-click,
" o: open & close node
" O: recursively open node
" t: open in new tab
" T: open in new tab silently
" x: close parent of node
" X: close all child nodes of
"    current node recursively
" middle-click,
" e: explore selected dir
"
" ----------------------------
" Bookmark table mappings~
" double-click,
" o: open bookmark
" t: open in new tab
" T: open in new tab silently
" D: delete bookmark
"
" ----------------------------
" Tree navigation mappings~
" P: go to root
" p: go to parent
" K: go to first child
" J: go to last child
" <C-j>: go to next sibling
" <C-k>: go to prev sibling
"
" ----------------------------
" Filesystem mappings~
" C: change tree root to the
"    selected dir
" u: move tree root up a dir
" U: move tree root up a dir
"    but leave old root open
" r: refresh cursor dir
" R: refresh current root
" m: Show menu
" cd:change the CWD to the
"    selected dir
" CD:change tree root to CWD
"
" ----------------------------
" Tree filtering mappings~
" I: hidden files (off)
" f: file filters (on)
" F: files (on)
" B: bookmarks (off)
"
" ----------------------------
" Custom mappings~
"
" ----------------------------
" Other mappings~
" q: Close the NERDTree window
" A: Zoom (maximize-minimize)
"    the NERDTree window
" ?: toggle help

Acknowledgements

GitHub - scrooloose/nerdtree: A tree explorer plugin for vim.

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Automating Tmux sessions with tmuxp

2018-03-24T22:05:00+01:00

It's more convenient to have all of this little details that you normally do day-to-day already pre-scripted - especially if you use it on a computer that you shutdown in a day-to-day fashion.

That's why I decided to improve my day-to-day blogging experience with tmux and use Tmuxp.

Tmuxp is a tmux session manager created with python. It uses tmuxinator and teamocil style as it says in README docs.

I'm amused how much features it contains!

To The Point

Installation of Tmuxp

As my personal preference I use pipsi to install some of the script-alike stuff - just like a tmuxp:

pipsi install tmuxp

You can also install tmuxp with pip (in a virtualenv if you prefer, or not, up to you):

pip install tmuxp

Usage of Tmuxp

Once you will create your first configuration, you can load it as a tmux session with:

tmuxp load yourtmuxconfig.yaml # or yourtmuxconfig.json

For the example below, it will create 4 panes that are tiled - meanig as a 4 equaly defined boxes.

BTW - did you know that if you have already one session with the same name as the one you are loading, tmuxp will ask you if you want to attach to that session?

That's very convenient! So instead of making tmux attach -t nameofsession you can actually use the tmux load yourconfig.json and it will attach after answering y to that created session.

Yaml and Json config files

So in Tmuxp you can actually use two type of files - yaml and json.

Simple example of that:

YAML:

session_name: 4-pane-split
windows:
- window_name: dev window
  layout: tiled
  shell_command_before:
    - cd ~/
  panes:
    - shell_command:
        - cd /var/log
        - ls -al | grep \.log
    - echo hello
    - echo hello
    - echo hello

JSON:

{
"windows": [
    {
        "panes": [
            {
            "shell_command": [
                "cd /var/log", 
                "ls -al | grep \\.log"
            ]
            }, 
            "echo hello",
            "echo hello",
            "echo hello"
        ], 
        "shell_command_before": [
            "cd ~/"
        ], 
        "layout": "tiled", 
        "window_name": "dev window"
    }
], 
"session_name": "4-pane-split"
}

So comparing those two - it feels like YAML is easier to use and less coding needed to have a working solution. But with my vim and python experience I find it less handy. It's probably because I have predefined strangly tab to always insert as 4 spaces no matter what type of file I'm editing (python requirements). Yeah I will fix that someday, but for now it makes my experience with YAML files less pleasant so I'd better be off with JSON as a personal preference.

Configuration

Tmuxp configuration consists of sessions and windows and panes.

Sessions and windows

Your sessions are at the bottom.

Creating new session with one pane and only session name using JSON file:

{
"windows":[ ],
"session_name": "4-pane-split"
}

Multiple Windows

{
"windows":[
    {
        "panes": []
    },{
        "panes": []
    }
],
"session_name": "2-pane-split"
}

Multiple Windows with focus specified

{
"windows":[
    {
        "panes": [
            "echo 'first pane'"
        ]
    },
    {
        "panes": [
            "echo 'second pane'"
        ],
        "focus": true
    },
    {
        "panes": [
            "echo 'third pane'"
        ]
    }
],
"session_name": "focus-pane"
}

Configuration options that you can use

shell_command_before - this is used before making call on each of the pane's scripts.
window_name - this is a name for each of the windows that are within this session.
suppress_history - to override default comamnd suppression.
window_index - if you want to change the default sorting and placement of windows at your session
start_directory - you can globally use this variable as well as locally for specific pane scripts and nesting one to another as you can see in my configuration
focus - option to make one of the windows and panes to be focused after loading config - unfortunatelly it did not worked for me and as listed in this issue, i'm not the only one having this bug.
options and global_options- this contains changing settings (per window or globally) for:
- automatic-rename - you can set if you want your panes and windows to autotically change name or not
- default-shell - you can set your default shell that will be run on panes
- default-command - you can set your default command on panes
- main-pane-height - you can set height of pane.

There is probably a lot more but that should be enough to make some sophisticated tmux session configuration for tmuxp.

Check out more at the example section in documentation of tmuxp

My personal usage for creating this blog

For now I decided create tmuxp config like that:

{
    "start_directory": "~/projects/anselmos_blog/",
    "windows": [
        {
            "window_index": 0,
            "window_name": "content",
            "start_directory": "./data/content/articles",
            "panes": [
                {
                   "shell_command": [
                       "vim ."
                    ],
                    "focus": true
                }
            ],
            "focus": true
        },
        {
            "window_index": 1,
            "window_name": "dev",
            "start_directory": "./data",
            "shell_command_before": [
                "cd pelican-plugins/pelican_link_to_title/",
                "sh runredis.sh",
                "cd -"
            ],
            "panes": [
                "pipenv run make devserver"
            ]
        },
        {
            "window_index": 2,
            "window_name": "publish",
            "shell_command_before": [
                "cd ~/projects/anselmos_blog/"
            ],
            "panes": [
                "echo 'make publish_ftp'"
            ]
        },
        {
            "window_index": 3,
            "window_name": "social",
            "shell_command_before": [
                "cd ~/projects/day_to_day_automation/blog/"
            ],
            "panes": [
                "echo 'pipenv run python get_post_summary.py ~/projects/anselmos_blog/data/content/articles/'" 
            ]
        }
    ], 
    "session_name": "blog"
}

This creates 4 panes: - one with vim opened at articles directory - second with pelican starting development-ready server that automatically detects changes in .md files. - third with echoing make publish_ftp for my convenience to have it already there. - last one that is for re-using the get_post_summary.py as seen in this article

Asciinema

4 pane split example:

2 empty panes (check it in a fullscreen to see bottom pane with tmux name of panes):

3 empty panes with focus on the middle one (check it in a fullscreen to see bottom pane with tmux name of panes):

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Python Django Framework examples of decorators

2018-03-23T18:55:00+01:00

To improve learning of decorators I've decided to find real-life examples of decorators in django framework.

To The Point

To be more precise in our adventure through decorators I decided to look for the most used python frameworks/libraries and find decorators that may be useful.

In Django documentation you can find View Decorators which explains view decorators. In the source-code of django you can find those decorator packages/modules:

django/contrib/auth/decorators.py
django/django/utils/decorators.py
django/views/decorators/:
- require_http_methods is a http-method decorator that uses @wraps - from which we learn that using @wraps at decorator inner-function prevents problem with passing __docs__ and __name__ of decorated function.
- require_GET is a derivation of require_http_methods which includes only GET method as a parameter to that function. - also it changes __doc__ of the function to have different docs than in require_http_methods - for more readability.

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Python Function Annotations

2018-03-22T09:20:00+01:00

Within PEP 3107 you can find the original PEP(Python Enhancement Proposals) for annotations of funcion parameters and return values.

This proposal that has been accepted is unfortunatelly only for Python3 and ongoing releases. Python2 out of the box backward compatibility is not possible.

Yet you can create some backward compatibility for those annotations if you need to.

Check out what this annotations are all about!

To The Point

Annotations

In other languages you have types. You expressly define your variable types, function return types and parameters types.

In Python 2 you could not do that (it would raise with SyntaxError) and frankly speaking you did not expect to do that.

In Python 3 you can use annotations to simulate defining function and variable types.

Here is example:

def greeting(name: str) -> str:
    return 'Hello ' + name

So basically you have created a function that in python2 would look like this:

def greeting(name):
    return 'Hello ' + name

But annotations gives you opportunity to check what type of output and parameter types you have within this function:

greeting.__annotations__

Unfortunatelly or fortunatelly it does not do anything else - So if you make annotations but you do not follow it - it will not raise it.

We can change that with decorator that will check that:

def decorator_annotation_for_return(function_wrapped):
    def wrapper(*args, **kwargs):
        annotations = function_wrapped.__annotations__
        output = function_wrapped(*args, **kwargs)
        returned_type = annotations.get('return')
        if returned_type:
            assert type(output) == returned_type

        return output
    return wrapper

Example of usage for this decorator:

@decorator_annotation_for_return
def greetings(name: str) -> str:
    return "Hello " + name

print(greetings("Anselmos"))

# This will fail:
@decorator_annotation_for_return
def greetings_failing(name: str) -> str:
    return 1

greetings_failing("A")

You can also create a parameter types checker with decorators, but in order for it to work properly you would need to use **kwargs - which means you need to expressively name arguments for function.

Check out example of that decorator:

def decorator_annotation_parameters(function_wrapped):
    def wrapper(*args, **kwargs):
        annotations = function_wrapped.__annotations__
        output = function_wrapped(*args, **kwargs)
        returned_type = annotations.get('return')
        for kwarg in kwargs:
            kwarg_type = annotations.get(kwarg)
            assert type(kwargs[kwarg]) == kwarg_type
        if returned_type:
            assert type(output) == returned_type

        return output
    return wrapper

@decorator_annotation_parameters
def greetings_parameter(name: str) -> str:
    return name


print(greetings_parameter(name=1))

It may need some tweaking to have better readiness but it works.

Also check out the suggested in pep Use Cases.

Snippets

def function(parameter: annotation_parameter_type) -> annotation_parameter_type:
    pass

def decorator_annotation_parameters(function_wrapped):
    def wrapper(*args, **kwargs):
        annotations = function_wrapped.__annotations__
        output = function_wrapped(*args, **kwargs)
        returned_type = annotations.get('return')
        for kwarg in kwargs:
            kwarg_type = annotations.get(kwarg)
            assert type(kwargs[kwarg]) == kwarg_type
        if returned_type:
            assert type(output) == returned_type

        return output
    return wrapper

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Creating RSS Feed links in pelican theme

2018-03-21T08:57:00+01:00

There are at least 2 options that I know how to add RSS link to pelican-bootstrap3 theme.

To The Point

Adding RSS

First things first - if you have not yet setup your RSS at pelican with pelican-bootstrap3, you can with this in your configuration:

AUTHOR_FEED_ATOM = None # here you can put something like 'feeds/%s.author.xml'
AUTHOR_FEED_RSS = None # here you can put something like 'feeds/%s.author.xml'
FEED_ALL_ATOM = 'feeds/all.atom.xml'
TAG_FEED_ATOM = 'feeds/%s.tag.atom.xml'
CATEGORY_FEED_ATOM = 'feeds/%s.atom.xml'

RSS at the top menu of side.

You obviously can add RSS link on the top menu.

To do that add to your pelicanconf.py or publishconf.py in MENUITEMS list item like that:

MENUITEMS = [
    ['RSS', "feeds/all.atom.xml"]
]

RSS in the blogroll link

In the "blogroll" links:

LINKS = (
    ('RSS', "feeds/all.atom.xml")
)

Tweaking top-menu theme of pelican-bootstrap3

As suggested in this pelican blog setup - I decided to also tweak my top-menu theme items with icons.

To do this I've changed MENUITEMS generation at template/base.html with this:

{% for title, link, icon in MENUITEMS %}
    <li><a href="{{ link }}"><i class="fa {{icon}} fa-lg"></i>{{ title }}</a></li>
{% endfor %}

The MENUITEMS now looks like this:

MENUITEMS = [
    ['RSS', "feeds/all.atom.xml", 'fa-rss']
]

So from now on I can add one of FontAwesome icons into my menu-items with ease.

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Accelerating pelican plugin execution

2018-03-20T23:30:00+01:00

In previous post I have made some quick fix with redis that makes re-loading links a bit faster.

While I was searching for some plugins that I could use in my blog, I've found this plugin source code which uses content_object_init signal. Let's check if using this will make our plugin even faster.

To The Point

To be precise and have reliable performance checks let's first change our debugging project and add more articles with links to it.

Source code of debugging plugin for performance can be found at this branch.

Comparison

Without plugin

Took 0.14 second.
With previous plugin (redis-aware)

Took 0.17-0.20 second.
With content_object_init change.

First time:

Took ~10 second.

Second time(and next):

Took ~0.20 second.

Source Code of changed plugin

# -*- coding: utf-8 -*-
""" This is a main script for pelican_link_to_title """
from pelican import signals
from bs4 import BeautifulSoup
import urllib


def link_to_title_plugin(generator):
    "Link_to_Title plugin "
    article_ahreftag= {}
    for article in generator.articles:
        soup = BeautifulSoup(article._content, 'html.parser')
        ahref_tag = soup.find_all('ahref')
        if ahref_tag:
            article_ahreftag[article] = (ahref_tag, soup)

    for article, (p_tags, soup) in article_ahreftag.items():
        for tag in p_tags:
            url_page = tag.string
            if url_page:
                if "http://" in url_page or "https://" in url_page:
                    tag.name = "a"
                    tag.string = read_page(url_page)
                    tag.attrs = {"href": url_page}
            else:
                continue
        article._content = str(soup).decode("utf-8")

def read_page(url_page):
    import redis
    redconn = redis.Redis(host='localhost', port=6379, db=0)
    found = redconn.get(url_page)
    if not found:
        r = urllib.urlopen(url_page).read()
        soup = BeautifulSoup(r , "html.parser")
        title = soup.find("title").string
        redconn.set(url_page, title)
        return title
    else:
        return found


def content_object_init(instance):
    if instance._content is not None:
        content = instance._content
        soup = BeautifulSoup(content, "html5lib")

        for ctbl in soup.find_all('ahref'):
            url_page = ctbl.contents[0]
            if url_page:
                if "http://" in url_page or "https://" in url_page:
                    ctbl.name = "a"
            try:
                ctbl.string = read_page(url_page)
            except:
                pass
            ctbl.attrs = {"href": url_page}
        instance._content = soup.decode()
            # If beautiful soup appended html tags.
        if instance._content.startswith('<html>'):
            instance._content = instance._content[12:-14]

def register():
    """ Registers Plugin """
    signals.content_object_init.connect(content_object_init)
    # signals.article_generator_finalized.connect(link_to_title_plugin)

Effects

Well as I see the efects are not so much different.

I will need to check if updating urllib into something with parallelism will make it better.

Acknowledgements

Auto-promotion

ToThePoint

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Debugging slowness of pelican-link-to-title plugin

2018-03-19T22:20:00+01:00

In the Making pelican-link-to-title plugin article I've created a neat script that uses <ahref>link</ahref> and transforms this to <a href="link">TITLE OF LINK</a>. Today I'm going to analyze what I've done wrong while creating this plugin. Check out.

To The Point

Slowness of plugin

How can you analyze slowness of plugin?

First let's test how much time it will take to generate html content without plugin:

git clone --recursive-modules https://github.com/anselmos/debug_pelican_plugins # or --recursive if you have git < 2.13
cd debug_pelican_plugins
pipenv shell
make html

The last command shows how much time it took to generate pages.

To be able to check performance of plugin I need to add a bit more than only one link.

Let's create a content that will have at least 5 different page links.

Title: first
Date: 2018-03-19 20:30
Status: published
Author: Witkowski Bartosz
Slug: first
# first

# some topic for first


<ahref>http://bbc.com/</ahref>
<ahref>https://stackoverflow.com/</ahref>
<ahref>https://github.com/</ahref>
<ahref>http://abc.go.com/</ahref>
<ahref>https://www.quora.com</ahref>
<ahref>https://www.thegoodwebguide.co.uk/lifestyle/website-reviews/best-sites-news-information/best-alternative-news-sites/13301</ahref>
<ahref>http://wakatime.com/</ahref>
<ahref>http://twitter.com/</ahref>
<ahref>https://www.facebook.com/AnselmosBlog/</ahref>
<ahref>https://tomato-timer.com/</ahref>

And now let's find out how much time it takes to generate content for this article without plugin, and with plugin:

without: ~ 0.15 s. with: ~ 10-12 s ...

And it's after a bit of optimizations with looping.

The biggest bottleneck is the urllib and obtaining page-source with it. There is solution with httplib2 which I'm going to cover in next episode.

This time let's focus on making a type of caching for this requests.

Let's create a redis cache with link-title elements.

Redis

Redis is a key-value pair database server (no-sql). It's fast and reliable. Let's make a query with it in our plugin:

Starting redis from Docker

To start Redis you can install it on system. I prefer use docker instead:

docker run --name pelican-redis -v $(pwd)/redis:/data -d redis redis-server --appendonly yes

and the plugin looks like this:

# -*- coding: utf-8 -*-
""" This is a main script for pelican_link_to_title """
from pelican import signals
from bs4 import BeautifulSoup
import urllib


def link_to_title_plugin(generator):
    "Link_to_Title plugin "
    article_ahreftag= {}
    for article in generator.articles:
        soup = BeautifulSoup(article._content, 'html.parser')
        ahref_tag = soup.find_all('ahref')
        if ahref_tag:
            article_ahreftag[article] = (ahref_tag, soup)

    for article, (p_tags, soup) in article_ahreftag.items():
        for tag in p_tags:
            url_page = tag.string
            if url_page:
                if "http://" in url_page or "https://" in url_page:
                    tag.name = "a"
                    tag.string = read_page(url_page)
                    tag.attrs = {"href": url_page}
            else:
                continue
        article._content = str(soup).decode("utf-8")

def read_page(url_page):
    import redis
    redconn = redis.Redis(host='localhost', port=6379, db=0)
    found = redconn.get(url_page)
    if not found:
        r = urllib.urlopen(url_page).read()
        soup = BeautifulSoup(r , "html.parser")
        title = soup.find("title").string
        redconn.set(url_page, title)
        return title
    else:
        return found

def register():
    """ Registers Plugin """
    signals.article_generator_finalized.connect(link_to_title_plugin)

This makes rendering again at the ~0.15 s. :)

This has limitations - first time when links are obtained by urllib it will take ~10-12 seconds.

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Gitgutter Vim plugin

2018-03-18T07:30:00+01:00

GitGutter plugin lets you manage your file changes within git-repository with vim by adding signs at the left side to the line numbers.

Also, you can :

Move between git-hunks (changes chunks)
Undo your git-hunks
See all hunks in a vim-split.

To The Point

Installation

To install gitgutter, I'd suggest using Vundle with adding this to your .vimrc config file:

Plugin 'airblade/vim-gitgutter'

Then install usually with:

vim +PluginInstall +qall

For more information about installation, check out installation guide provided by the author of the plugin.

Usage

To use this plugin all you need to do is first a add file to your git repository and then start making changes in it.

Please be adviced that Vim has an updatetime flag that I suggest you tweak from 4000ms to (as I checked) around 250ms. Changing to 250ms is a bit better and less clunky compared to what author suggests with 100ms.

To make that change, add this to your vim-config .vimrc file:

set updatetime=250

Turning gitgutter off

If you don't like how gitgutter works you may uninstall this plugin or if it's only for some of the files you can disable this plugin with vim-command:

:GitGutterDisable

Turning gitgutter on

:GitGutterEnable

GitGutter toggle on/off

:GitGutterToggle

Jumping to next change/chunk

Use ]c for next and [c for previous.

To customise key-mapping, use this:

nmap ]h <Plug>GitGutterNextHunk
nmap [h <Plug>GitGutterPrevHunk

Undoing your hunks

You can undo hanks with <Leader>hu or re-map key-combination with this:

nmap <Leader>hu <Plug>GitGutterUndoHunk

Preview hunk

You can preview hunk with <Leader>hp or re-map key-combination with this:

nmap <Leader>hv <Plug>GitGutterPreviewHunk

Snippets

GitGutter Vim-Commands and KeyCombinations:

Disable: :GitGutterDisable
Enable: :GitGutterEnable
Toggle: :GitGutterToggle
NextHunk: ]h ( re-mapped from ]c )
PrevHunk: [h ( re-mapped from [c )
UndoHunk: <Leader>hu
PreviewHunk: <Leader>hv ( re-mapped from <Leader>hp )

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

How to use pyperclip

2018-03-17T00:00:00+01:00

Pyperclip is a python module that enables managing our system clipboard. It is a cross-platform module.

To The Point

Installation of pyperclip

System Dependency

On Windows you don't need to install any additional software.

On Linux you need to install either xclip or xsel, but the recommended is xclip:

sudo apt-get install xclip

Installation

Python module can be installed with simple pip command :

pip install pyperclip

Or via pipenv :

pipenv install pyperclip

Using pyperclip

That's pretty easy, commands are :

pyperclip.copy - for making a copy of data to clipboard (as a parameter)
pyperclip.paste - for pasting from clipboard data in python.

Examples of usage

I'm going to use pyperclip as a system for copying summaries from blog-posts.

Check out this script that I'm going to use to make my day-to-day easier and more automated:

import pyperclip
import argparse
YOUR_DOMAIN = "http://witkowskibartosz.com/blog/"

def get_summary(post_file_markdown):
    SLUG_TAG = "Slug: "
    SUMMARY_TAG = "Summary: "
    summary = ""
    slug = ""
    with open(post_file_markdown, 'r') as filemarkdown:
        for line in filemarkdown.read().split("\n"):
            if SLUG_TAG in line:
                slug = line.split(SLUG_TAG)[1]
            if SUMMARY_TAG in line:
                summary = line.split(SUMMARY_TAG)[1]

    return (slug, summary)

def make_social_text_and_copy(slug, summary):
    text = summary + " : "+ YOUR_DOMAIN + slug + ".html"
    pyperclip.copy(text)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Arguments of this script')
    parser.add_argument('markdownfile', metavar='N', type=str, nargs='+')
    args = parser.parse_args()
    data = get_summary(args.markdownfile[0])
    make_social_text_and_copy(*data)

This basically reads md file, looks for SLUG and SUMMARY and makes a social text that is being copied to clipboard.

To use this script, you need to add path to md file as argument to python script like that:

pipenv run python get_post_summary.py ~/your_path_to_md_file.md

Snippets

Simulates CTRL+C (copy):

pyperclip.copy('some text that goes to clipboard')

Simulates CTRL+V (paste):

pyperclip.paste()

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

Using markings and vim-signature plugin

2018-03-16T08:00:00+01:00

You can look at the project-page of this plugin to see it in action github-repo.

This plugin allows to see markings of file at the left bar before line-numbers.

To The Point

Usages of vim markings and vim-signature

Vim Marks:

You can jump right to the place where you left writing your code/blog-post/whatever.
You can sign a place that is important in project you are working on to remember about something in there (as opposite to #FIXME/ #TODO which lefts data in code)

Vim Signature plugin

Allows to see where the markings are set instead of making call to :marks

How to install plugin

Usually with Vundle add this to your vimrc file:

Plugin 'kshenoy/vim-signature'

Between those vim-config lines:

call vundle#begin()

call vundle#end()            " required

Now you need to install your plugin with:

vim +InstallPlugin +qall

And re-bash to new shell with installed vim-plugin.

Using Markings and Vim-Signature plugin

Vim Signature will display all markings at the left side.

To use markings you should follow this awesome guide

Making new mark: Press m and some letter like a so : press: ma on the line that you want to place an a marking.
Jumping to the a mark with keys: \a`
Listing all markings with vim-command: :marks

Snippets

Posting here some of the marking command combinations from Vim.Wikia.com as a reference:

Command         Description
ma              set mark a at current cursor location
'a              jump to line of mark a (first non-blank character in line)
`a              jump to position (line and column) of mark a
d'a             delete from current line to line of mark a
d`a             delete from current cursor position to position of mark a
c'a             change text from current line to line of mark a
y`a             yank text to unnamed buffer from cursor to position of mark a
:marks          list all the current marks
:marks aB       list marks a, B



]'              jump to next line with a lowercase mark
['              jump to previous line with a lowercase mark
]`              jump to next lowercase mark
[`              jump to previous lowercase mark

The above commands take a count. For example, 5]` jumps to the fifth mark after the cursor. 

Command         Description
`.              jump to position where last change occurred in current buffer
`"              jump to position where last exited current buffer
`0              jump to position in last file edited (when exited Vim)
`1              like `0 but the previous file (also `2 etc)
''              jump back (to line in current buffer where jumped from)
``              jump back (to position in current buffer where jumped from)
`[ or `]        jump to beginning/end of previously changed or yanked text
`< or `>        jump to beginning/end of last visual selection 


:delmarks a     delete mark a
:delmarks a-d   delete marks a, b, c, d
:delmarks abxy  delete marks a, b, x, y
:delmarks aA    delete marks a, A
:delmarks!      delete all lowercase marks for the current buffer (a-z)

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't - up to you.

Any suggestions what I should blog about? Post me a comment in the box below or poke me at Twitter: @anselmos88.

See you in the next episode! Cheers!

How to remove old Git branches

2018-03-15T20:30:00+01:00

It's common to need to remove old git branches. It's almost a day to day task that the most of developers do.

To The Point

Before you start removing your branches you should checkout to master with :

git checkout origin/master

Removing a local branch

git branch -D name-of-branch

Removing a branch on remote

BEAWARE This will make a request to git server to remove branch on remote!

git push origin :name-of-branch-to-remove-on-git-server

Removing local branches stale on remote

With fetching

git fetch -p

Only removing

git remote origin prune

Snippets

git fetch -p

git remote origin prune

BONUS

After some digging I've found there is a better git-command that can be used to prune - and it's git-prune - unfortunatelly it's available from 2.16.0 version, and as I've found out with :

git --version

I was using a 1.9.1 version :/

Let's change that:

Update git to latest version

sudo add-apt-repository -y ppa:git-core/ppa
sudo apt-get update
sudo apt-get install -y git

Now for Ubuntu 14.04 I have a 2.16.2 version.

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Selenium Firefox profile for Automation

2018-03-14T18:30:00+01:00

Did you know that you can actually use Firefox Profile in your Selenium automation? It's even quite simple as you will figure out in this article.

To The Point

Creating a new Firefox Profile

Some command lines for firefox are described at this link, but not all of them are available at linux. I checked available commands with:

firefox --help

To create your profile:

firefox -P newprofile

It will popup with a ProfileManager that is not available at normal Firefox run. You can create in this window new profile.

I did so and It displayed a window with path to profile (which btw you can change) - it looks similar to this:

~/.mozilla/firefox/YOURUNIQUENAME.NAMEYOUENTERED

You cannot copy this path while creating profile, but when you go to path with profiles:

~/.mozilla/firefox/

You should see it.

We can do some changes on our profile with starting the same command

firefox -P newprofile

then clicking on "Start Firefox" while selecting profile that we want.

Starting Selenium with firefox profile:

    profile_path = '/home/anselmos/.mozilla/firefox/your_profile_path_directory_enter_here'
    profile = webdriver.FirefoxProfile(profile_path)
    driver = webdriver.Firefox(profile)

From now you can use a working profiled Selenium Firefox.

Snippets

    profile_path = '/home/anselmos/.mozilla/firefox/your_profile_path_directory_enter_here'
    profile = webdriver.FirefoxProfile(profile_path)
    driver = webdriver.Firefox(profile)

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Automation for Procrastination

2018-03-13T21:05:00+01:00

Today at 6 am I've found out that I'm procrastinating for about 1 hour. When I realized that I decided to stop and check my Nozbe account with tasks that I could use for blog-topic. You know based on the article name what topic I've found.

You may ask - how you want to use automation in fight with procrastination?

I've found out that using Pomodoro Technique and special for each of us music gives me more focus than I could ever thought. That's why I'm going to make an automation that will start both of this elements at my browser to get more focused right-away.

I'll use one of the Pomodoro Technique timers and Spotify playlist to play music.

Without further ado, check out this selenium automation I will create!

To The Point

Selenium for Spotify web-player.

First you need to have a spotify playlist that you personally like to listen to get more focused. I have some of my preferred playlists, one of them is Blue Stahli- Antisleep vol1.

Let's create an automation which for now will open this link :

from page_objects import PageObject, PageElement
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

class SpotifyPlaylist(PageObject):

    """Spotify Playlist Page for Selenium Page Object Pattern"""
    album_uri = "https://open.spotify.com/album/"
    playlist_uri = "https://open.spotify.com/playlist/"
    login_uri = "https://spotify.com/login/"

    username = PageElement(css='input[id="login-username"]')
    password = PageElement(css='input[id="login-password"]')
    login_btn = PageElement(css='button[id="g-recaptcha-button"]')

    def open_playlist(self, playlist_id):
        self.get(self.playlist_uri + playlist_id)

    def open_album(self, album_id):
        self.get(self.album_uri + album_id)

    def login(self, username, password):
        self.get(self.login_uri)
        self.username = username
        self.password = password
        self.login_btn.click()

Using it with:

    driver = webdriver.Firefox()
    spotify_page = SpotifyPlaylist(driver)
    username = "heregoesyourusername"
    password = "heregoesyourpassword"
    spotify_page.login(username, password)
    spotify_page.open_album('56QpDqnTzXqMn3Jjxv73lz')

It will automatically login and go to album.

Taking into accunt that Spotify is not DRM Free, it needs to have Play DRM settings.

We will cover this in tomorrow topic how to do that because solution is not simplest as it seems.

For now lets move to our second page that we need to make less Procrastination.

Pomodoro APP

I have found that using some pomodoro-technique timer makes me a lot more productive. Let's make a script that will automatically start a pomodoro timer - because that's what I'm currently doing manually.

from page_objects import PageObject, PageElement
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

class PomodoroPage(PageObject):

    """Spotify Playlist Page for Selenium Page Object Pattern"""
    main_uri = "https://tomato-timer.com/"

    timer_start = PageElement(css='button[id="timer_start"]')

    def start_pomodoro(self):
        self.get(self.main_uri)
        self.timer_start.click()

Now we need to mix them together. Unfortunatelly I've found that making a new tab in browser has some strange issues. To overcome it you can use a javascript code.

The script looks like this:

    driver.execute_script('''window.open("https://google.com/", "_blank");''')

The final script is available at Day-to-Day Automation github repo directory

Snippets

Script for creating new tab at firefox selenium driver:

    driver.execute_script('''window.open("https://google.com/", "_blank");''')

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Python decorators vs inheritance

2018-03-12T05:40:00+01:00

In previous article about class-based decorators for classes I gave you example how one can create decorator.

Those decorators have similarities to inheritance between classes.

I will review pros and cons for class-inheritance and decorator(function/class based).

To The Point

Decorator example:

from timeit import default_timer as timer
class TimingDecorator(object):
    def __init__(self, klas):
        self.klas = klas
        self.org_calculation1 = self.klas.calculation1
        self.klas.calculation1 = self.calculation1

    def __call__(self, arg=None):
        return self.klas.__call__()

    def calculation1(self, *args, **kwargs):
        start = timer()
        self.org_calculation1(self.klas, *args, **kwargs)
        end = timer()
        print(end - start)

@TimingDecorator
class MyFancyClassTest():

    def calculation1(self, a, b):
        from time import sleep
        # i'm adding a sleep - as a test case if counter works.
        sleep(2)
        return (a+b*2)/(3*a)

fancy_object = MyFancyClassTest()
fancy_object.calculation1(50, 215)

Pros:

you can make an extension for class-methods without inheritance
you could mix inheritance with decorators
you still have the same class-object.

Cons:

all methods needs to be initialized and re-assigned at init
for each method you need to create another method in decorator (not flexible enough)
decorator is limited to this class methods - can not be reused at different class with different methods.

Function decorator example:

def counter_print(function_name):
    def wrapper(*args, **kwargs):
        start = timer()
        output_of_function = function_name(*args, **kwargs)
        end = timer()
        print(end - start)
        return output_of_function

    return wrapper

class MyFancyClassTest():

    @counter_print
    def calculation1(self, a, b):
        from time import sleep
        # i'm adding a sleep - as a test case if counter works.
        sleep(2)
        return (a+b*2)/(3*a)

fancy_object = MyFancyClassTest()
fancy_object.calculation1(50, 215)

Pros:

you still have the same class-object
you could can mix inheritance with decorators
you don't need to re-create assignment for predefined class-methods
your decorator is not limited to only this specific class. It is re-usable.

Cons:

Not advice to use this solution if you have a lot of code to make. It can get messy.

Inheritance example

class MyFancyClassTest():

    def calculation1(self, a, b):
        from time import sleep
        # i'm adding a sleep - as a test case if counter works.
        sleep(2)
        return (a+b*2)/(3*a)

class TimerMyFancyClassTest(MyFancyClassTest):

    def calculation1(self, *args, **kwargs):
        start = timer()
        output = super().calculation1(*args, **kwargs)
        end = timer()
        print(end - start)
        return output



fancy_object = TimerMyFancyClassTest()
fancy_object.calculation1(50, 215)

Pros:

your extended class does the same as decorator (prints timing for method)

Cons:

You need to create code that extends main code
It is not re-usable/adoptable to other class-methods
you endup with different object and comparison is a bit more complex

MIX (inheritance at decorators class)

from timeit import default_timer as timer

class MetaClassDecorator(object):
    """ A meta class decorator that forces creating methods from decorator-appended class."""
    def __init__(self, klas):
        self.klas = klas
        for klass_method in self.get_klas_methods():
            exec('self.org_'+klass_method+' = self.klas.'+klass_method)
            exec('self.klas.'+klass_method+' = self.'+klass_method)

    def get_klas_methods(self):
        """ based on https://stackoverflow.com/a/34452 """
        all_methods = [
            method_name for method_name in dir(self.klas)
            if callable(getattr(self.klas, method_name)) and method_name[0:2] != "__"
        ]
        return all_methods

    def __call__(self, arg=None):
        return self.klas.__call__()

class FancyTestDecor(MetaClassDecorator):
    """ This decorator forces 2 methods - fancy_method and test1 """

    def fancy_method(self, *args, **kwargs):
        start = timer()
        self.org_fancy_method(self.klas, *args, **kwargs)
        end = timer()
        print(end - start)

    def test1(self, *args, **kwargs):
        start = timer()
        self.org_test1(self.klas, *args, **kwargs)
        end = timer()
        print(end - start)


@FancyTestDecor
class MyFancyClassTest():

    def fancy_method(self):
        print("fancy print from fancy method")
    def test1(self):
        print("Ttest11")

fancy_object = MyFancyClassTest()
fancy_object.fancy_method()
fancy_object.test1()

Pros:

you re-use pre-existing architecture for decorator.
you create only class with methods (reusing meta-decorator-class)
you can create a more complex solutions as decorators for classes.

Cons:

All methods from MyFancyClassTest if any found, needs to be added and re-assigned to fields, because of exec for all methods that does not start with "__" private functions/methods - check docs for 2.7 python.
You create a non-reusable code just as in class-based decorator (unless you find other class that will have only those methods) (unless you find solution to replace the exec problem and re-assign methods without manually changing them at __init__)

Snippets

class MetaClassDecorator(object):
    """ A meta class decorator that forces creating methods from decorator-appended class."""
    def __init__(self, klas):
        self.klas = klas
        for klass_method in self.get_klas_methods():
            exec('self.org_'+klass_method+' = self.klas.'+klass_method)
            exec('self.klas.'+klass_method+' = self.'+klass_method)

    def get_klas_methods(self):
        """ based on https://stackoverflow.com/a/34452 """
        all_methods = [
            method_name for method_name in dir(self.klas)
            if callable(getattr(self.klas, method_name)) and method_name[0:2] != "__"
        ]
        return all_methods

    def __call__(self, arg=None):
        return self.klas.__call__()

class FancyTestDecor(MetaClassDecorator):
    """ This decorator counts timing of fancy_method method """

    def fancy_method(self, *args, **kwargs):
        start = timer()
        self.org_fancy_method(self.klas, *args, **kwargs)
        end = timer()
        print(end - start)

Acknowledgements

Auto-promotion

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Python class based decorator for classes

2018-03-11T10:45:00+01:00

As I've shown you in previous episode about parameterizing decorators you can create a class-based decorator.

This time let's create a derivation class-based decorator that is intended for classes.

To The Point

First, we create our class that will use decorator.

class MyFancyClassTest():

    def fancy_method(self):
        print("fancy print from fancy method")

The decorator itself with class:

class DecoratorForClasses(object):
    def __init__(self, klas):
        self.klas = klas
        self.org_fancy_method = self.klas.fancy_method
        self.klas.fancy_method = self.fancy_method
        print("init")

    def __call__(self, arg=None):
        print("call")
        return self.klas.__call__()

    def fancy_method(self):
        print("before pancy method (at decorator)")
        self.org_fancy_method(self.klas)
        print("after pancy method (at decorator)")


@DecoratorForClasses
class MyFancyClassTest():

    def fancy_method(self):
        print("fancy print from fancy method")

fancy_object = MyFancyClassTest()
fancy_object.fancy_method()

After analizing this, I see a lot of similarities with python-inherintance in it.

Yet since decorator does not have any other inheritance, but instead can "extend" capabilities of classes, it is still useful.

Snippets

class DecoratorForClasses(object):
    def __init__(self, klas):
        self.klas = klas
        self.org_method1 = self.klas.method1
        self.klas.method1 = self.method1

    def __call__(self, *args, **kwargs):
        return self.klas.__call__(*args, **kwargs)

    def method1(self, *args, **kwargs):
        print("method1_decorator")
        output = self.org_method1(self.klas, *args, **kwargs)
        return output


@DecoratorForClasses
class MyFancyClassTest():

    def method1(self):
        print("fancy print from fancy method")

fancy_object = MyFancyClassTest()
fancy_object.method1()

Acknowledgements

Auto-Promotion

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Python decorators with parameters

2018-03-10T16:30:00+01:00

Python Decorators have the ability to be parameterized.

The main reason is to have more ability to change some behaviour of function/class in a more elegant DRY way.

To The Point

You can create your own decorator that will be a class.

This time we will use that, to make our code a bit more simplier and readable.

This example is a more/less based on this stackoverflow answer:

class BeforeAfterDecorator(object):
    def __init__(self, before_func, after_func, which_return = 0):
        self.before_func = before_func
        self.after_func = after_func
        self.which_return = which_return

    def __call__(self, original_func):

        def wrappee( *args, **kwargs):
            output_before = self.before_func()
            output = original_func(*args, **kwargs)
            output_after = self.after_func()
            if self.which_return == 1:
                return output_before
            elif self.which_return == 2:
                return output_after
            else:
                return output
        return wrappee

def before_function():
    print("this is a more complex function that will be invoked before function")
    return "before"
def after_function():
    print("this is a more complex function that will be invoked after function")
    return "after"

@BeforeAfterDecorator(before_function, after_function, 1)
def make_all_return_before(a,b,c):
    print( 'in bar',a,b,c )
    return "method"

@BeforeAfterDecorator(before_function, after_function, 2)
def make_all_return_after(a,b,c):
    print( 'in bar',a,b,c )
    return "method"

@BeforeAfterDecorator(before_function, after_function)
def make_all_return_output(a,b,c):
    print( 'in bar',a,b,c )
    return "method"


first_output = make_all_return_before('x','y','z')
print("function ended with return: ", first_output)
second_output = make_all_return_output('x','y','z')
print("function ended with return: ", second_output)
third_output = make_all_return_after('x','y','z')
print("function ended with return: ", third_output)

As you can see in script above, this decorator (BeforeAfterDecorator) has ability to take other functions that will be invoked before decorated function and after. Also it has third parameter that decides which output will be taken.

Where you could use that?

For example at the Fabric which can be used for deployment process. Check out my posts about using fabric.

At the stackoverflow answer you can find alternative solution to make meta-decorator that uses triple-nest function. I suggest you to check it.

This decorator looks like this:

def parametrized(dec):
    def layer(*args, **kwargs):
        def repl(f):
            return dec(f, *args, **kwargs)
        return repl
    return layer

Snippets

class ParameterizedDecorator(object):
    def __init__(self, your_decorator_parameter):
        self.your_decorator_parameter = your_decorator_parameter

    def __call__(self, original_func):
        def wrappee( *args, **kwargs):
            print(self.your_decorator_parameter)
            output = original_func(*args, **kwargs)
            return output
        return wrappee

Acknowledgements

Autopromotion

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Python decorators for Classes

2018-03-09T07:55:00+01:00

Classes in Python can be extended by inheritance. In Python 3 ability to make a class-decorator was added per PEP request.

Check out that you can make a type of inheritance from decorator to add something to method like in this example

To The Point

How to make your own class decorator? First you need a class definition so let's create one :

class MyFancyClass:
    def __call__(self):
        print("You were calling sir?!")

fancy_object = MyFancyClass()
fancy_object()

Now we can create our class decorator:

def fancy_class_decorator(cls):
    old_init = cls.__init__
    def new_init(self, *args, **kwargs):
        old_init(self, *args, **kwargs)
        print("here goes your additional code to init!")
    cls.__init__ = new_init
    return cls

And using decorator :

@fancy_class_decorator
class MyFancyClass:
    def __call__(self):
        print("You were calling sir?!")

fancy_object = MyFancyClass()
fancy_object()

While I was researching topic I've found that making a class-only fields that are not initialized within constructor can lead to a bad results.

Check out this:

def fancy_class_decorator(cls):
    old_init = cls.__init__
    def new_init(self, *args, **kwargs):
        self.onevar = "fancy_decorator_before"
        self.before = "fancy_decorator_before"

        old_init(self, *args, **kwargs)

        self.onevar = "fancy_decorator_after"
        self.after = "fancy_decorator_after"

    cls.__init__ = new_init
    return cls

@fancy_class_decorator
class MyFancyClass:
    onevar = "class"
    before = "class"
    after = "class"

    def __call__(self):
        print("onevar", self.onevar)
        print("before", self.before)
        print("after", self.after)

fancy_object = MyFancyClass()
fancy_object()

And then This:

def fancy_class_decorator(cls):
    old_init = cls.__init__
    def new_init(self, *args, **kwargs):
        self.onevar = "fancy_decorator_before"
        self.before = "fancy_decorator_before"

        old_init(self, *args, **kwargs)

        self.onevar = "fancy_decorator_after"
        self.after = "fancy_decorator_after"

    cls.__init__ = new_init
    return cls

@fancy_class_decorator
class MyFancyClass:
    def __init__(self):
        self.onevar = "class"
        self.before = "class"
        self.after = "class"

    def __call__(self):
        print("onevar", self.onevar)
        print("before", self.before)
        print("after", self.after)

fancy_object = MyFancyClass()
fancy_object()

First example presents situation where you create a class-based fields which are not initialized within constructor of class.

Because we are making a decorator-inheritance over constructor, it automatically changes fields data.

Snippets

Replace old_method with method you want to make decorator-inheritance on.

def class_decorator(cls):
    old_method = cls.method
    def new_method(self, *args, **kwargs):
        old_method(self, *args, **kwargs)

    cls.old_method = new_method
    return cls

Acknowledgements

Auto-promotion:

Found related to post:

Not related, but also time-worth:

Pilcrow

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Python decorators for function

2018-03-08T08:15:00+01:00

What are python decorators? How can you create your own function decorator? Check out!

What are python decorators

Python decorators are a wrappers around the function/method/class that can do actions before/after some function/method/class is invoked.

To make that happen, you re-use function as a parameter. Basically you create a private function that contains other function as a parameter. Sounds like not a trivial thing? Don't be scared, it's pretty simple.

As a reference think about a profiler that can measure how much time your function took. With wrapper it's easy-peasy.

To The Point

To create your first decorator, first you create a function wrapper:

def wrapper(*args, **kwargs):
    return function_that_you_take_as_argument()

But wait, what is the function_that_you_take_as_argument ? It's exactly what it says - a function that is taken with decorator as argument.

To make a working decorator you also need to make an outer-function around it:

def decorator_name(function_that_you_take_as_argument):
    def wrapper(*args, **kwargs):
        return function_that_you_take_as_argument()
    return wrapper

@decorator_name
def function_that_you_take_as_argument()
    print("test")

This decorator does nothing at this time. But if you add output to variable and make statements before and after like that:

def decorator_name(function_that_you_take_as_argument):
    def wrapper(*args, **kwargs):
        print("before:")
        output =function_that_you_take_as_argument()
        print("after:")
        return output
    return wrapper

@decorator_name
def your_function_this_name_is_different_then_in_wrapper_argument()
    print("test")

Then your function call with decorator will:

first print before:
make function - print test
print after -when function has done it's executement.

Snippets

def decorator_name(function_wrapped):
    def wrapper(*args, **kwargs):
        output = function_wrapped(*args, **kwargs)
        return output
    return wrapper

Asciinema

Example from asciinema

#! /usr/bin/env python
# -*- coding: utf-8 -*-
# vim:fenc=utf-8
#
# Copyright © 2018 [Anselmos](github.com/anselmos) <anselmos@users.noreply.github.com>
#
# Distributed under terms of the MIT license.

"""
Decorators example
"""

def decorator_name(function_name):
    """ Here you put your function """
    def wrapper(*args, **kwargs):

        print("before invoking function")
        output_of_function = function_name(*args, **kwargs)
        print("after invoking function")
        return output_of_function
    return wrapper

@decorator_name
def your_function_that_will_use_decorator():
    """TODO: Docstring for your_function_that_will_use_decorator.
    :returns: TODO

    """
    print("Test")

your_function_that_will_use_decorator()

Acknowledgements

Autopromotion:

Found while research of the topic

List of Python build-in Decorators

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Pipenv run vs Pipenv shell

2018-03-07T07:25:00+01:00

Pipenv has a run and shell, but which one should I use ? That's the same question I've asked myself yesterday. Check out my brief summary about this!

Pipenv shell

Pipenv shell is used mainly as a run it now, and leave.

It will create an active environemnt almost the same way as with source env/bin/activate, but it will also do it within a new shell. That way you cannot use previous commands from history.

There is even a bug about that but the conclusion is that it's not pipenv issue, rather pew problem with sub-shell.

As it's using a sub-shell, it may not be advised to use it within your deployment/management.

For example if you want to make a Dockerfile with Pipenv, you should not use pipenv shell because it will not be activated for the next RUN command or any other command that you may put with && signs.

You could use some type of sub-processing but that's far beyond easy and clean - so as using pipenv --venv that gives you opportunity to reuse underlying virtualenv (by giving you place where is the v-env that you can activate manually).

Pipenv run

Pipenv run is the main purpose for running next python commands. It will automatically activate environment for that command and execute anything you will put to it.

For example to run some specific python file that uses dependency you will:

pipenv run python main.py

That will activate env and run python main.py in the env.

Conclusion

As posted by Dan Ryan(techalchemy) in this issue comment - you can should use pipenv run for 'heavy lifting' and pipenv shell for day-to-day convenience of dev tasks.

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Locking Pipfile with Pipenv

2018-03-06T23:08:00+01:00

Today I've got a problem with pipenv - Locking your Pipfile with pipenv. Check out how easy it is!

In the previous article about pipfile I've presented some key commands how to use this powerful yet effortless python management tool.

This time let's dive a bit further!

The case with problem.

Pipenv is a great tool, but in some cases it may fail in one or another way.

Today I had a bit of failure of this tool. But it was not pipenv fault. It was a more Pip and System dependencies problem.

To the point - I installed psycopg2 with Pipenv but forgot that it needs postgres-dev package at the system. It failed.

While it failed, Pipenv made a call and did not create Pipfile.lock - a lock file.

So immediatelly after I've realized what just happend, I've created a docker image ( Ha!) with a pipenv(1.0) and postgresql system dependency(2.0).

Then I've found that I could easily make it with pipenv lock which was incredible.

Pipenv locking

Pipfile.lock will be generated whenever you make :

pipenv install --dev
pipenv install [package-name]
pipenv uninstall [package-name]

One of the times when your Pipfile.lock may not be updated is - when the Pipfile has been changed, i.e. when you made some git - merging process and changed versions of packages in Pipfile.

Then you may need to refresh your Pipfile.lock because that changes are not reflected in Pipfile.lock.

To do that you will use pipenv lock - isn't it easy ?.

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Your coding timer with Wakatime

2018-03-05T19:07:00+01:00

Have you ever wonder how much time you spend on your IDE and focused work like programming/writing code? Today I'm going to check if wakatime is solution to find that!

Wakatime

A Wakatime is a time-management tool. It automates process of gathering how much time you have spend on coding and typing text within your editor.

Today I've decided finally to check if this is a great tool or not.

Plugins

This time management tool has a list of editor integrations that works.

That's why I've decided to use it - because in that list there are two tools without which I don't see my programmers day - vim and pycharm.

Let's check out how to integrate them!

Creating your account on Wakatime.

First you need to create your account on Wakatime. It will be needed for having API-KEY pasted into config.

Vim plugin

As usually you will use Vundle to install plugin.

Add this to your .vimrc file:

Plugin 'wakatime/vim-wakatime'

And then:

vim +PluginInstall +qall

Plugin should automatically be installed.

After installation and re-accessing your shell and vim, you should be asked for API-KEY. Paste the one you have been granted at here.

This key will be saved in your config file at :

~/.wakatime.cfg

Pycharm Plugin

In Pycharm, as usually, go to Settings and Plugins, find plugin via "Browse Repositories" and install.

After installation, you should restart your Pycharm, if you are not asked for API-KEY - no worries.

You can change it at 'Tools' -> 'WakaTime Settings' and "API Key".

Copy the key from ~/.wakatime.cfg file and you should be ready to go.

Statistics!

After a while of coding I've found that it gathers huge amount of data which then is presented to user at Dashboard which you can check at this image

There are also a "LeaderBoards" where you can start a competition to code a bit more.

Excluding files

What if i don't want to have some files/project to be visible at Waka?

Check out how to setup it with config file at faq.

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you in the next episode! Cheers!

Python snippets in Vim with Ultisnips

2018-03-04T18:40:00+01:00

Snippets in all type of editors are very usefull beings that every bloger/programmer should start using. Especially if using ViM. Check out what solution I've found that supports snippets for vim!

S0-E30/E30

UltiSnips

There is a great snippets solution for Vim UltiSnips created iniatially by Holger Rapp, that is also recommended by Django Team at Vim Hints.

Let's check out what they are all about!

Installing UltiSnips

First of all, you need to install your UltiSnips.

As usually I'm recommending Vundle that makes installing of vim-plugins much easier.

Let's do it. Add this line in your .vimrc file:

" Track the engine.
Plugin 'SirVer/ultisnips'

" Snippets are separated from the engine. Add this if you want them:
Plugin 'honza/vim-snippets'

" Trigger configuration. Do not use <tab> if you use https://github.com/Valloric/YouCompleteMe.
let g:UltiSnipsExpandTrigger="<tab>"
let g:UltiSnipsJumpForwardTrigger="<c-j>"
let g:UltiSnipsJumpBackwardTrigger="<c-k>"

" If you want :UltiSnipsEdit to split your window.
let g:UltiSnipsEditSplit="vertical"

Now as usually make installation of this plugin with:

vim +PluginInstall +qall

UltiSnips Templates

When you follow steps above, you will have already predefined templates from honza/vim-snippets repository

You can check those templates in your vim-bundle directory at:

ls -al ~/.vim/bundle/vim-snippets/UltiSnips/

I personally checked the source code for preexisting python snippet to know which snippets are available.

Using UltiSnips

When we copy-pasted changes to vimrc file, there was a "UltiSnipsExpandTrigger", "UltiSnipsJumpForwardTrigger" and "UltiSnipsJumpBackwardTrigger".

They are in fact a binding to which key-stroke special action should be used. It means, that using in our case a "Tab" after writing name of the snippet i.e. :

deff<TAB>

will make use of preexisting deff snippet and paste it to our code/text.

Those snippets have special moving triggers, which applies to move forward/backward within snippet elements to be changed.

Where to go now?

Holdger Rapp (author of UltiSnipps) YT video UltiSnips Playlist, EP1, EP2, EP3, EP4
VimCasts Videos - You can also check how to use this UltiSnips with videos from vimcasts.
Gitter communicator for UltiSnips - a place where community paste they problems and solves them. Check at SirVer/Ultisnips

I used those videos and it helped me to learn how to use this powerful feature.

Acknowledgements

Thanks!

I'd like to address a special thanks for #miroburn for challenging me in making a 30-day blog-posting! It was a great experience!

See you in the next episode soon! Cheers!

Vim Templates are a life saver

2018-03-03T17:00:00+01:00

Today when my 30daychallenge almost finishes (tomorrow is the last day) I feel that I could not make it without reusing previous posts for new posts. Check out how to automate it!

S0-E29/E30

The problem

As you may or may not find - most of my posts are currently made with a ViM editor.

It can be a bit painful sometimes, especially when it goes with day-to-day blog-posting. So to be able to deliver content I was cheating a little - to be more precise - I used a Copy-and-Paste method. That means I was making a copy of the previous post and replaced it with current content.

A fun fact is that this post is also written like that. That's a bit silly, so let's change it, shall we?

Vim Templates

There is a plugin called vim-templates by Adrian Perez that gives us the opportunity to make a life of a blogger and a programmer a bit easier.

You can find the project on GitHub repo - Vim-Templates.

How to install

The most comfortable way will be by using Vundle Vim Plugin Manager. That's the one I'm using, and that's why I'll cover how to do it this way.

Add Plugin to your .vimrc file where you usually place your plugins:

Plugin 'aperezdc/vim-template'

And now install it with :

vim +PluginInstall +qall

That's it. You should have your plugin installed.

You should now get to new shell env to get your plugin running.

How to use it?

Start a new file in vim, and if your extension is in templates directory, then you will get automatically generated content from that template.

Of course, you may not want to use automatically generated content by this plugin, then remove auto-cmd within .vimrc:

let g:templates_no_autocmd = 1

Then to use this plugin all you need to do is use the vim command:

:Template *.&lt;here-goes-template-extension&gt;

i.e. for blog markdown file, I would use:

:Template *.md

Or for Python file:

:Template *.py

BE AWARE of using :Template

Making a vim-command :Template *.py in an already filled file, will append content from template to your vim-buffer at the top.

Where are the templates?

Template files are located usually within directory:

ls ~/.vim/bundle/vim-template/templates/

Out of the box you have :

ls -l ~/.vim/bundle/vim-template/templates/ | sort | uniq -c | wc -l

so about 50 file-templates.

Can I extend them?

Yes, you can extend them! Tweak them to make what you want them to do.

There are some unique characters in templates that make the whole process of automation even a bit further.

I.e. the python template uses something like this:

#! /usr/bin/env python
# -*- coding: utf-8 -*-
# vim:fenc=utf-8
#
# Copyright © %YEAR% %USER% &lt;%MAIL%&gt;
#
# Distributed under terms of the %LICENSE% license.

"""
%HERE%
"""

Where you find special characters like:

%YEAR% - a current year.
%USER% - a name from your computer user. Change it within .vimrc with : let g:username = 'yourusername'
%MAIL% - email is a combination of your name on the computer and a host-name. This can be change within .vimrc with : let g:email = 'anselmos@users.noreply.github.com'
%LICENSE% - which by default creates a "MIT" sign. You can change this within .vimrc with : let g:license = 'yourslicensename'
%HERE% - will give nothing, but ensure that vim-cursor will get placed in the place where this sign is.

For more knowledge

You can check documentation with vim using :

:help templates.txt

Also, check out in browser documentation of this plugin.

I will surely check it whenever I will need to change something with the template.

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Callable in python - do you even use it?

2018-03-02T23:00:00+01:00

Today let's find out what is the purpose of the callable function and in which versions of python it does not work.

S0-E28/E30 :)

Callable

A callable is a function that checks other methods/functions/classes to find out if that particular element has a __call__ implemented.

Since variables like int or boolean will not have that, making :

callable(1)

callable(True)

Will output a "False". Meaning that those elements are not callable.

But let's check if an object will return true.

class ASimpleClass():
    pass

element1 = ASimpleClass()
callable(element1)

No. It does not - why? because it's an Object, not a Callable.

Now Let's find out if a class will be a callable:

class Test():
    pass
callable(Test)

And sure thing, it does.

Also for functions it is valid:

def function1():
    pass
assert callable(function)
assert not callable(function())

Is there a situation where you would not recommand callable?

Yes. In fact, callable was removed between 3.0 to 3.2. In 3.2 + it was brought back.

So I would check if we are using 3.0 to 3.2 python. If yes, then I would not use callable. I would then check with code like that:

isinstance(f, collections.Callable)

Acknowledgement

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

One year blog Anniversary - Why do I still blog?

2018-03-01T22:30:00+01:00

Today is the day when this Blog has it's One Year Anniversary. Exactly One year ago I decided to start blogging. Check out why after one year I still blog.

S0-E27/E30 :)

What was my first article?

You can look back using page-navigator on a list of posts and find out that my first blog post was How to create own blog which give a lot of input on making your own Blog! Even I am really a fan of it - because of amount of knowledge I've put into that particular article.

What I've prepared for 1st year blog anniversary

I've added two plugins:

Series - a great plugin that enables the creation of series of posts. Configured by blogger. In my blog, you can check one of the #30dayMiroBurnChallenge posts
Related-posts - also a great plugin that enables making relations between posts that contains similar tags. It works automatically but can you can override its default behaviour. I have already enabled this plugin, Just go to the end of this posts and you will have five related posts.

I decided finally to clean up a bit my markdown directory with posts and move files from one "bin" into individual directories i.e. projects.

I do hope you those plugins will help you find some interesting articles.

Now to the question.

Why do I still blog

I do tend to have a break from now to then. But even if I do - I come back. I believe it's because of those arguments:

I feel achievement every time I put an article on the blog. (even if it's a straightforward tutorial) - sometimes I come-back to that post for knowledge.
Each time I blog, I try to find an interesting thing that I'm currently working on or that fascinates me. That helps me to move forward with my knowledge and experience.
Sharing knowledge with others! - You have a friend that's likely to have a problem that you already solved? Why not share it on your blog or a StackOverflow?
Making a post about something gives you an opportunity to learn that thing better - Why ? - Because you need to undestand beforehand what are you are blogging about - and that makes a type of "Rubber duck debugging" process in your mind - which alternatively improves the learning process.

That's about it - that's what drives my blogging experience.

How about you? When do you plan to start your Blog ?

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Should I use has_key or in for dict at Python?

2018-02-28T18:30:00+01:00

Today a very brief topic about using has_key and my journey on finding that it does not work on Python3. Check out!

S0-E26/E30 :)

Python Dictionaries

Let's say you have a dict like this:

test_dict = {"testkey": "testvalue", "TestKey2": "TEstkeyVal2"}

And you want to know that there is a testkey in this dictionary.

At python2+ you could do that :

test_dict = {"testkey": "testvalue", "TestKey2": "TEstkeyVal2"}
assert test_dict.has_key("testkey")

And the assertion would not raise.

But in Python3+ you have an error:

assert test_dict.has_key("testkey")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  AttributeError: 'dict' object has no attribute 'has_key'

That's because has_key has been removed from python3.

What should I do?

Instead of using has_key use in.

assert "testkey" in test_dict

Should I use dict.keys() to check then?

Short version - you should not - why ? because you may find yourself someday in a performance issue that is hard to trace.

So don't make use of something like this:

if "key1" in dict1.keys()

Instead use:

if "key1" in dict1

Check this stackoverflow comment about performance

or in this hidden code that I've copy-pasted so you don't need to go to SO:

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to use Pipenv

2018-02-27T20:30:00+01:00

Have you noticed that recommendation for Managing Application Dependency at python.org has been changed? Apparently our python community should use now Pipenv. But how to? Let's check out!

S0-E25/E30 :)

Pipenv

The Pipenv has been created by Kenneth Reitz - who complained at his blog in this post - A better pip workflow - about pip workflow - and then created this tool.

You can check Readme for pypenv at their github repository to find out what problems it solves :) I encourage you to do that !

Let's get started!

Installing Pipenv

So first things first - What do we need for Pipenv ?

pip - you can find out how to get pip here - but for most of the *nix users - you should already have pip packaged with your system.

Then:

pip install pipenv

Adding a shell-completion!

Bash

Put command below into your .bashrc or .bash_profile:

eval "$(pipenv --completion)"

Basic usage

Let's make some day-to-day tasks with pipenv:

Create new project using python3.6

First let's make a project with :

mkdir pipenv_test_project
cd pipenv_test_project
git init .
git commit --allow-empty -m "Init"
git remote add origin git@github.com:anselmos/pipenv_test_project.git
git push origin HEAD:master

Initialize pipenv at project

And now initialize pipenv at the pipenv_test_project:

pipenv --python 3.6

Activating pipenv at project

pipenv shell

REMEMBER - you need to be in the root path of the project.

Install all dependences including dev:

pipenv install --dev

This will create a Pipfile.lock.

Let's install some dependency i.e. selenium

pipenv install selenium

And your Pipfile and Pipfile.lock has been changed.

Installing old version of pelican and upgrading

pipenv install pelican==3.6.0

Upgrading to latest version of pelican

pipenv install pelican

Which will automatically check version and install the latest - 3.7.1 ! Yeah.

Uninstalling pelican

pipenv uninstall pelican

The `pipenv_test_project`

The project for this article is available here.

That was an introduction to pipenv - if you need a more advanced stuff - check documentation or pipenv --man

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Making a FullScreen Screenshot with Firefox Node on Selenium Hub

2018-02-26T20:00:00+01:00

Today we are going to make a fullscreen screenshot at Selenium Hub using Firefox node. Check out how to fix the issue.

This article is a followup of Grammarly Integration - intended for fixing fullscreen issue - you can check there how to make a screenshot.

S0-E24/E30 :)

Full Screen Screenshot within Selenium.

There is a problem with latest versions of Firefox - and from what I've heard only FF v47.0.1 works properly.

So Let's check out how to make a fullscreen screenshot with this version!

Firefox Node and Selenium Hub.

After quick research of the topic I've found version of Selenium Hub and Firefox Node docker images that makes fullscreen possible.

Version of Node and Hub

Version of the hub and node in docker-images is 2.53.1.

Beaware that you need to pull Hub and Node. Using Latest Hub and previous version of Firefox Node will not work.

Running Firefox Node and Hub

docker rm -f selenium-hub
docker run -d -p 4444:4444 --name selenium-hub selenium/hub:2.53.1
docker rm -f selenium-node-firefox
docker run -d --link selenium-hub:hub --name selenium-node-firefox -v /dev/shm:/dev/shm selenium/node-firefox:2.53.1

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Making Pelican Link To Title Plugin

2018-02-25T23:00:00+01:00

Today we are going to dive into Pelican, and it's plugins :) We are going to make a plugin that will transform only a link into an HTML link with a description consisting of the page title.

S0-E23/E30 :)

Pelican Plugins

In this blog we have already created pelican-plugin, you can check pelican-plugin tag.

What type of signals pelican has to offer?

Well, that's the thing, they are not first-class documented things.

For making them a bit more available and easier to debug, I've created debug_pelican_signals repository at github that contains all signals which you can now connect to Pycharm or pdb and debug. The methods are visible at signalsdebugger.py of debug_pelican_signals repo.

Making a Pelican Link To Title

Now let's make a pelican link to title plugin that will convert all links into links with a title.

The plugin will be called 'Pelican Link to Title'. Its source code is available at github.

Code itself:

# -*- coding: utf-8 -*-
""" This is a main script for pelican_link_to_title """
from pelican import signals
from bs4 import BeautifulSoup
import urllib


def link_to_title_plugin(generator):
    "Link_to_Title plugin "
    article_ptags = {}
    for article in generator.articles:
        soup = BeautifulSoup(article._content, 'html.parser')
        p_tags = soup.find_all('ahref')
        for tag in p_tags:
            url_page = tag.string
            if url_page:
                if "http://" in url_page or "https://" in url_page:
                    tag.name = "a"
                    tag.string = read_page(url_page)
                    tag.attrs = {"href": url_page}
        article._content = str(soup).decode("utf-8")

def read_page(page):
    r = urllib.urlopen(page).read()
    soup = BeautifulSoup(r , "html.parser")
    return soup.find("title").string

def register():
    """ Registers Plugin """
    signals.article_generator_finalized.connect(link_to_title_plugin)

To use the plugin, all you need to do is add into your PLUGINS in pelicanconf a pelican-link-to-title and put in the article something like this:

<ahref>http://witkowskibartosz.com/blog</ahref>

and this "ahref" will be retransformed into:

<a href="http://witkowskibartosz.com/blog/">Anselmos Blog</a>

Which means, it goes into this page, takes the title of the page and puts into a-href string data.

Acknowledgements

Web Scraping with Beautiful Soup

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Making Elasticsearch Your Blog Search Engine.

2018-02-24T21:35:00+01:00

Do you have a blog, that lacks search engine that could find in content of your article where you wrote about this and that ? Let's check if Elasticsearch can help with it. Check out!

S0-E22/E30 :)

Elasticsearch store your pelican posts

This blog uses Pelican engine that uses either a reStructuredText or Markdown files which then are transformed into a html with some theme.

Because of that, you can't have a search engine on html files online. Fortunately you can use Elasticsearch, push your rst/md Files and search for specific words in it :)

Let's check how to achieve this.

Making a Search within your own .md files.

If you are so lucky to have some text files, you can start off. For my prof of concept purpose we'll use github repo from pelican blog.

Let's make a script that would put data from md/rst files into Elasticsearch.

Ofcourse let's don't repeat, and reuse what we have learned from yesterday's elasticsearch python client

Listing files with extension in python

import os
from glob import glob
def list_files(path, fileextension):
    return [y for x in os.walk(path) for y in glob(os.path.join(x[0], '*.{}'.format(fileextension)))]

Putting file context into elasticsearch

def import_rst_files():
    all_rst_files = list_files(mypath, "rst")
    docs = []
    for rst_file in all_rst_files:
        client = Elasticsearch('localhost')
        # Readfile content:
        content = read_file_content(rst_file)
        doc = {

            "_index": "blogpost-{}".format(datetime.datetime.now().strftime("%Y-%m-%d")),
            "_type": "blogpost",
            "_id": rst_file,
            "_source": {
                "author": "PelicanblogAuthors",
                "content": content,
            }
        }
        docs.append(doc)
    helpers.bulk(client, docs)

def read_file_content(filename):
    return open(filename, 'r').read().decode('utf-8')

Making a very trivial searching

def search_in_elastic(phrase_to_search="blog"):
    from elasticsearch_dsl import Search
    client = Elasticsearch('localhost')
    s = Search(using=client).query('match', content=phrase_to_search)
    response = s.execute()
    return response

def print_found(search_response):
    print pprint.pformat(search_response.hits.hits)

Execution

import_rst_files()
found = search_in_elastic()
print_found(found)

This will list files, read them, put into elasticsearch and then output all found blog-posts with 'blog' phrase.

Bonus

You can find the code listed above in this repository.

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Utilizing Elasticsearch Python Client

2018-02-23T23:00:00+01:00

Elasticsearch is a great search engine, but using JSON and curl does not fit python. Fortunately there are two libraries that you can use - and in today's article I'll focus on that :) Check out!

S0-E21/E30 :)

Elasticsearch python wrappers

How to start with Python Wrapper for Elasticsearch engine?

That's pretty easy. First, we need to setup our elasticsearch engine by using a jar file and use java, or for Docker to handle this heavy topic - which you can find in my previous post.

When you do that, we can start tinkering with it using two python wrappers.

First one is the Official one and it's called Low Level because it uses simple api that makes just requests with json data (converted from python-dicts). This one is called Elasticsearch-py.

Second one is a bit more complex but also very user-friendly and rather easy to use and learn. It uses a different approach. Reusing official library, makes use of python's lambdas making a very data-flow-friendly api.

Let's check them out!

Elasticsearch-py

Dependencies

The only one is the library itself - you can install it with:

pip install elasticsearch

Example

This is an example of data that you could put in a blog-post:

import datetime
from elasticsearch import Elasticsearch
client = Elasticsearch('localhost')
blog_post = {
    "author": "Anselmos",
    "date": datetime.datetime.now().strftime("%Y-%m-%d"),
    "content": "A new Blog-Post Content!",
}
res = client.index(index="test-index", doc_type='blogpost', id=1, body=blog_post)
print res

What do you think what will be output of this script?


{u'_type': u'blogpost', u'_seq_no': 0, u'_shards': {u'successful': 1, u'failed': 0, u'total': 2}, u'_index': u'test-index', u'_version': 1, u'_primary_term': 1, u'result': u'created', u'_id': u'1'}

And the Searching :

from elasticsearch import Elasticsearch
client = Elasticsearch('localhost')
res = client.get(index="test-index", doc_type='blogpost', id=1)
print res

How to you think, will it output my data?


{u'_type': u'blogpost', u'_source': {u'date': u'2018-02-23', u'content': u'A new Blog-Post Content!', u'author': u'Anselmos'}, u'_index': u'test-index', u'_version': 3, u'found': True, u'_id': u'1'}

Elasticsearch-dsl-py

Dependencies

The only one is the library itself - you can install it with:

pip install elasticsearch-dsl

Search Query!

This library has a more advanced features that can come in handy especially in a advanced search with filtering.

To make a better use of this library, let's make a list of data for searching:

from elasticsearch import Elasticsearch
import random
import datetime
from elasticsearch import helpers
client = Elasticsearch('localhost')

tag = ['blog', 'anselmos', 'elk', 'elastic', 'elasticsearch', 'elasticstack', 'elasticsearch-py', 'elasticsearch-dsl']
author = ['Anselmos', 'Bartosz Witkowski']
docs = [
    {

        "_index": "blogpost-{}".format(datetime.datetime.now().strftime("%Y-%m-%d")),
        "_type": "blogpost",
        "_id": x,
        "_source": {
            "author": author[random.randrange(0, len(author))]
            "date": datetime.datetime.now().strftime("%Y-%m-%d"),
            "content": "A new Blog-Post Content!",
            "tag": tag[random.randrange(0, len(tag))]
        }
    }
    for x in range(10)
]
helpers.bulk(client, docs)

That's an elasticsearch-py code that makes 10 blogposts with the same content, but different tags (randomized ).

Let's say, we wanted to know which tag have blog and author equals Anselmos.

with low-level api this would be:

from elasticsearch import Elasticsearch
import json
client = Elasticsearch('localhost')
queries = []
queries.append(
    {"query": {"bool": {"should": [{"match": {"tag": {"query": "blog"}}}]}}}
)
queries.append(
    {"query": {"bool": {"should": [{"match": {"author": {"query": "Anselmos"}}}]}}}
)
request = ''
for each in queries:
    request += '%s \n' % json.dumps(each)
res = client.msearch(index="blogpost-2018-02-23", doc_type='blogpost', body=queries)
print(res)

And how about hour High Level ?

Check-out

from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search
client = Elasticsearch('localhost')

s = Search(using=client).query('match', tag='blog').query('match', author='Anselmos')
response = s.execute()
print response

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to start with Elastic ELK Stack

2018-02-22T06:50:00+01:00

Today I'm going to present you an ELK - Elasticsearch stack solution and how to start with it :) Checkout :)

S0-E20/E30 :)

Elasticsearch - what is it?

Elasticsearch is a tool - or rather a stack of tools called ELK which combines at least 3 apps: Elasticsearch, Logstash and Kibana.

Logstash is just a propagator of data to searcher-elasticsearch which is visualised within Kibana :) That's a big shortcut of what ELK stack is.

Details and information about that are widely spread all over the internet, just check firstly here - at documentation

Elasticsearch ELK Stack with docker.

Elasticsearch

Based on tutorial there is an easy docker-run command for elasticsearch:

docker run -p 9200:9200 -p 9300:9300 --name elasticsearch -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.2.2

Logstash

docker pull docker.elastic.co/logstash/logstash:6.2.2
docker pull docker.elastic.co/logstash/logstash-oss:6.2.2

Without any volumens and configuration:

docker run --rm -it --link elasticsearch:localhost docker.elastic.co/logstash/logstash:6.2.2

And with only one config file:

docker run --rm -it -v ~/settings/logstash.yml:/usr/share/logstash/config/logstash.yml docker.elastic.co/logstash/logstash:6.2.2

Kibana

Run without configuration (just basic):

docker pull docker.elastic.co/kibana/kibana:6.2.2
docker run --rm -it --link elasticsearch:localhost -p 5601:5601 docker.elastic.co/kibana/kibana:6.2.2

A Bonus!

I've decided to make a simple github repo which intend is to use it as a starter for anyone looking into ELK first time.

Enjoy it :)

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to integrate Travis with Selenium Tests using Docker

2018-02-21T17:00:00+01:00

Integration of Travis with Selenium - is it possible? How would you do that? Checkout!

S0-E19/E30 :)

Travis and Docker

If you don't already know, I'm a big fan of the docker, and how the service has impacted developers society as such that each developer can take any new fancy-pancy tool and use it without having to get bored about the configuration of new server.

You may ask yourself - how is that related to Travis and Selenium ? Well... I'm going to try to use Travis and Docker Selenium image to run those tests :)

But did you knew there is possibility to make use of Docker within Travis?

Until yesterday I didn't know. But when I've found I've thought to myself - I have to check it!

So Let's check how to begin Travis-docker integration first shall we?

Travis file.

Travis as you can check in documentation should contain few key things. First of all:

language: python

Building a Python Project with Travis gives you a more advanced look at possibilities for Python-language project.

So a simple .travis.yml will look like this:

language: python
python:
    - "2.7"
    - "3.6"
# command to install dependencies
install:
 - pip install -r requirements.txt
# command to run tests
script:
 - pytest # or py.test for Python versions 3.5 and below

And docker will need to be handled like this As described here :

language: python
python:
    - "2.7"

services:
    - docker

before_install:
    - docker pull selenium/node-firefox:3.8.1-francium
    - docker pull selenium/hub:3.8.1-francium
    - docker run -d -p 4444:4444 --name selenium-hub selenium/hub:3.8.1-francium
    - docker run -d --link selenium-hub:hub --name selenium-node-firefox -v /dev/shm:/dev/shm selenium/node-firefox:3.8.1-francium

# command to install dependencies
install:
    - pip install -r requirements.txt
script:
    - python tests.py

Tests on Grammarly Selenium Automation Project!

So for our testing, I've used our Grammarly mini project to check if Travis will use it with Selenium-IDE and will use tests.

And it DID! checkout result for python2.7 here

Adjustment for python3.6

I've also found that pip did not setup accurately project with requirements.txt because those requirements were for python2.7. Let's fix this by changing a bit requirements.txt and check if tests on Travis will work :)

argparse==1.2.1
beautifulsoup4==4.6.0
page-objects==1.1.0
selenium==3.9.0
wsgiref

As you can see I've removed specific number of version for wsgiref library because it's the one that failed this travis job

This commit adds this fix and adds a Build-Status from Travis to README.md

The results are here

And as you can see this fix did not fix the problem :( So I've decided to remove a wsgiref library from requirements and test again.

Unfortunately as you can see, it also failed :) Because I forgot about very appealing redundant printing that was coded in python2.

So I got rid of it. Now take a look at this travis build. Taking into consideration that tests are not properly coded (using Python's time.sleep instead of Selenium Wait) sometimes This builds will fail -until I'll finally fix the problem :)

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Grammarly Integration for Pelican part3!

2018-02-20T22:00:00+01:00

Today more adventures about Grammarly and scraping data from checking-results.

S0-E18/E30 :)

Grammarly results scraping continues

After yesterday's article about possibility to scrape some results from Grammarly checking, let's now make a more advanced scraping :)

Let's make an example with more issues, so we can experiment with BeautifulSoap and scrape more with it.

I've found the "Demo document" that is presented within first usage of Grammarly, contains a lot of issues to present features of Grammarly. So let's use it :)

Scraping:

    def tests_all_in_one(self):
        from selenium import webdriver
        self.driver = webdriver.Firefox()
        filename = "demo_document.txt"
        demo_data_text = None
        with open(filename, 'r+') as f:
            demo_data_text = f.read().decode("utf-8")
        page_login = GrammarlyLogin(self.driver)
        page_login.make_login('za2217279@mvrht.net', 'test123')
        page_new_doc = GrammarlyNewDocument(self.driver)
        page_new_doc.make_new_document("")
        page_doc = GrammarlyDocument(self.driver)
        page_doc.put_title("DEMO DATA TEXT")
        page_doc.put_text(demo_data_text)
        page_source = GrammarlyDocument(self.driver)
        actual_source = page_source.get_page_source()
        scraper = DocumentScraper(actual_source)
        found_issues = scraper.find_all_issues()
        assert len(found_issues) == 14
        issues_by_type = scraper.return_issues_by_type()
        assert len(issues_by_type) == 2
        assert u'_ed4374-plainTextTitle' in issues_by_type
        assert u'_ed4374-titleReplacement' in issues_by_type
        assert len(issues_by_type['_ed4374-plainTextTitle']) == 3
        assert len(issues_by_type['_ed4374-titleReplacement']) == 11

And the source of Document Scraper:

from bs4 import BeautifulSoup

class DocumentScraper(object):

    def __init__(self, html_source):
        # self.html_source = html_source
        self.bs = BeautifulSoup(html_source, "html.parser")

    def get_issue_div(self):
        # DIV with class=_adbfa1e6-editor-page-cardsCol
        return self.bs.find('div', {'class': '_adbfa1e6-editor-page-cardsCol'})

    def get_all_warnings(self):
        return self.get_issue_div().contents

    def get_all_warnings_texts(self):
        return [element.text for element in  self.get_all_warnings()]

    def iterate_over_warnings(self):
        for innerelement in self.get_all_warnings():
            print innerelement.text

    def find_all_issues(self):
        return self.bs.findAll('div', {'class': '_ed4374-title'})

    def return_issues_by_type(self):
        issues = self.find_all_issues()
        output = {}
        for issue in issues:
            key = issue.contents[0].attrs['class'][0]
            try:
                output[key].append(issue.contents[0].contents)
            except KeyError:
                output[key] = [issue.contents[0].contents]

        return output

Where demo_document.txt contains "Demo Document" from Grammarly.

This gives you and example of how one can make our previous automation usable and transform it to something more.

Selenium Screenshots for better understanding Grammarly found issues.

Taking into account that there is no Selenium PDF generator - let's at least make a screenshot of page so we can then know where exactly those issues are:

        self.driver.save_screenshot('grammarly_checks.png')
        self.driver.get_screenshot_as_file('grammarly_checks2.png')

The only problem I've found with this is that it does not make a full-page screenshot. I remember that within Java that trick worked and you could make a full-web-page screenshot (and the webbrowser).

I might comeback to this :)

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Grammarly Integration for Pelican part2!

2018-02-19T23:55:00+01:00

Today let's focus on gathering results from Grammarly document checking in our mini-automation-project :)

This article is a part of Grammarly Selenium Pelican Automation Mini project

S0-E17/E30 :)

Grammarly Automation Gathering Report.

So Grammarly creates an report in a PDF. But the report in PDF is just a simple PDF creation of HTML file.

Why not disassemble that HTML page and get the results that we need instead of generating PDF and gathering only part of results?

Let's check if our assumptions meet with reality :)

So we have a text, lets for our example:

A simple
Multiline
text
That wraps
after max
2 words.

Put this into grammarly and gather HTML output it will generate.

Grammarly As I've found out will only generate pdf for premium users.

So let's focus on disassembling the html.

After gathering output from script:

    def test_simple_text_gather_html(self):
        """ This test is suppose to return html """
        page_login = GrammarlyLogin(self.driver)
        page_login.make_login('za2217279@mvrht.net', 'test123')
        page_new_doc = GrammarlyNewDocument(self.driver)
        page_new_doc.make_new_document("")
        page_doc = GrammarlyDocument(self.driver)
        text_to_put = "A simple \
            Multiline \
            text \
            That wraps \
            after max \
            2 words. \
        "
        page_doc.text = text_to_put
        self.sleep(10)
        print self.driver.page_source

I've found that there is an unique div called with css-class _adbfa1e6-editor-page-cardsCol That does not change from document to document. Maybe that's just a feature for react-app to know where to put the result of checking ? Either way - we have output!

But ... The output is not perfect. Data gathered in this way only gives a "shape" information - not exact details about where text is written in bad way - only information about specific text that is incorrect and the proposition for correct.

So instead of using Selenium for sculpturing I'll use python's html extractor - Beautiful Soup.

Let's make a source that will output html that then can be used within Beautiful Soup.

Now the GrammarlyDocument src looks like this:

class GrammarlyDocument(PageObject):

    title = PageElement(css='input[type="text"]')
    text = PageElement(id_='textarea')
    # This button below will only be visible for grammarly premium users.
    score_button = PageElement(css='span[class="_ff9902-score"]')
    download_pdf_btn = PageElement(css='div[class="_d0e45e-button _d0e45e-medium"]')

    def put_title(self, title):
        self.title = title

    def put_text(self, text):
        self.text = text
        time.sleep(10)

    def get_page_source(self):
        return self.w.page_source

And test:

    def test_get_page_source(self):
        page_login = GrammarlyLogin(self.driver)
        page_login.make_login('za2217279@mvrht.net', 'test123')
        page_new_doc = GrammarlyNewDocument(self.driver)
        page_new_doc.make_new_document("")
        page_doc = GrammarlyDocument(self.driver)
        text_to_put = "A simple \n\
            Multiline \n\
            text \n\
            That wraps \n\
            after max \n\
            2 words. \n\
        "
        page_doc.put_text(text_to_put)
        page_source = GrammarlyDocument(self.driver)
        actual_source = page_source.get_page_source()
        self.assertTrue("<html" in actual_source and "</html" in actual_source)

Yeah It's very silly test, but for now is sufficient.

Reverse parse HTML

Let's start with what the Beautiful Soup is.

It's a python html extractor that can be usefull when you want to scrape data from html. It's used widly with web-crawlers.

Now, let's make a DocumentScraper that will scrape data from html for us with at least the results data for now.

from bs4 import BeautifulSoup

class DocumentScraper(object):

    def __init__(self, html_source):
        # self.html_source = html_source
        self.bs = BeautifulSoup(html_source, "html.parser")

    def get_issue_div(self):
        # DIV with class=_adbfa1e6-editor-page-cardsCol
        return self.bs.find('div', {'class': '_adbfa1e6-editor-page-cardsCol'})

    def get_all_warnings(self):
        return self.get_issue_div().contents

    def get_all_warnings_texts(self):
        return [element.text for element in  self.get_all_warnings()]

    def iterate_over_warnings(self):
        for innerelement in self.get_all_warnings():
            print innerelement.text

And the simplest test on pre-downloaded file (output from selenium):

# -*- coding: utf-8 -*-
import unittest
from document_scraper import DocumentScraper

class GrammarlyScrapingTests(unittest.TestCase):

    def setUp(self):

        filename = "bs_output_test1.html"
        with open(filename, 'r+') as f:
            self.data_scrape1 = f.read()


    def test1(self):
        assert len(self.data_scrape1) == 21022
        scraper = DocumentScraper(self.data_scrape1)
        expected = [("Incorrect spacingwraps             after → wraps after".decode("utf-8"))]
        result = list(scraper.get_all_warnings_texts())
        assert result == expected

if __name__ == "__main__":
    unittest.main()

Part 3 !

There is going to be a part3 that will sum-up this mini-project and make this draft accessable.

So stay tuned :)

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Grammarly Integration for Pelican part1!

2018-02-18T16:30:00+01:00

There are no Grammarly applications that could deliver integration with at least command-line. So let's make selenium automation that will paste text into Grammarly documents!

S0-E16/E30 :)

Grammarly Integration

For now [mid-February 2018] Grammarly does not give their users API that you could use in integrations with some blogging engines, but instead, you can use their Android/iOS spelling keyboard on your mobile or use Web-browser.

There are integrations for Chrome as a plugin, Firefox addon and native in Windows and MacOS.

Since I'm a more into Linux - There is no real alternative than using their web-app.

Yes, they are excellent and I still use them. But for now, I don't have mobile-integration for my blogging experience, as Pelican is a static-site generator and I store files on my computer. I'm planning to make some automations, and this post is one of them :)

The best thing for now that I still do is manually copy-paste the actual text of blog-post and put it in Google-Keep and then use Grammarly Firefox Plugin. That seems a bit silly and not as handy as an Automation that could do that for me.

Selenium automation for Grammarly

So I said to myself - why not create Selenium automation that will at least put actual text of blog-post into Grammarly Document?

Grammarly-Selenium-Automation project init.

First, let's create a repository on GitHub called grammarly-selenium-automation. Then Let's make git-repo with:

mkdir grammarly-selenium-automation
cd grammarly-selenium-automation
git init .
git commit --allow-empty -m "Init"
wget https://www.gitignore.io/api/vim%2Cpython -O .gitignore
git add .gitignore
git commit -m "Adds gitignore"
git remote add origin git@github.com:anselmos/grammarly-selenium-automation.git
git push -u origin master

And now we have our repository available on github.

Now let's make actual selenium login-automation.

Grammarly Selenium login script.

For this script, I'll use one of my previous blog posts about making selenium-automation.

BTW - I've ofcourse forgot about making a virtualenv for using selenium :

cd grammarly-selenium-automation && virtualenv .env && source .env/bin/activate && pip install selenium && pip freeze > requirements.txt
git add requirements.txt
git commit -m "Adds pip requirements.txt"
git push origin HEAD:master

And now the script: login.py script

import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from grammarly_page import GrammarlyLogin
import time

class GrammarlyGeneralTest(unittest.TestCase):

    def setUp(self):
        # for dev I'll use only visible browser instead of no-gui one.
        # but for production, it will use the no-gui one.
        # self.driver = webdriver.Firefox()
        # TODO uncomment it when finished coding.
        self.driver = webdriver.Remote(
            command_executor='http://127.0.0.1:4444/wd/hub',
            desired_capabilities={'browserName': 'firefox', 'javascriptEnabled': True}
        )


    def test_grammarly_in_title(self):
        page_login = GrammarlyLogin(self.driver)
        self.driver.get(page_login.uri)
        assert "Grammarly" in self.driver.title

    def test_login(self):
        page_login = GrammarlyLogin(self.driver)
        page_login.make_login('za2217279@mvrht.net', 'test123')
        time.sleep(2)
        assert self.driver.current_url == "https://app.grammarly.com/"

    def tearDown(self):
        self.driver.close()

if __name__ == "__main__":
    unittest.main()

and the grammarly_page.py :

from page_objects import PageObject, PageElement

class GrammarlyLogin(PageObject):
    username = PageElement(css='input[type="email"]')
    password = PageElement(css='input[type="password"]')
    login = PageElement(css='button[type="submit"]')
    uri = 'https://www.grammarly.com'
    signin_uri = '/signin'

    def make_login(self, username, password):
        self.get(self.uri + self.signin_uri)
        self.username = username
        self.password = password
        self.login.click()

You can find it also at this repo tag: login-page-done

Grammarly Selenium adds new file:

Now ! :) to the fun-part :) Let's make a new document in Grammarly and push it to the grammarly with copying data.

While I was working on making new doc script, I've decided to change naming of the login.py file into tests.py - because I thought it will be more meaningful - since that's what it contains :)

Moving on :

Within grammarly_page.py I've added this:

class GrammarlyNewDocument(PageObject):
    uri = 'https://app.grammarly.com/'
    new_document = PageElement(css='div[role="button"]')

    def make_new_document(self, name=None):
        self.get(self.uri)
        self.new_document.click()
        time.sleep(2)


class GrammarlyDocument(PageObject):

    title = PageElement(css='input[type="text"]')
    text = PageElement(id_='textarea')

    def put_title(self, title):
        self.title = title

    def put_text(self, text):
        self.text = text

And on the tests.py I've added this:

    def test_make_new_doc(self):
        page_login = GrammarlyLogin(self.driver)
        page_login.make_login('za2217279@mvrht.net', 'test123')
        page_new_doc = GrammarlyNewDocument(self.driver)
        page_new_doc.make_new_document("TEST")
        assert "https://app.grammarly.com/docs/" in self.driver.current_url

    def test_change_title_on_doc(self):
        page_login = GrammarlyLogin(self.driver)
        page_login.make_login('za2217279@mvrht.net', 'test123')
        page_new_doc = GrammarlyNewDocument(self.driver)
        page_new_doc.make_new_document("")
        page_doc = GrammarlyDocument(self.driver)
        self.assert_title(page_doc)

    def test_change_text_on_doc(self):
        page_login = GrammarlyLogin(self.driver)
        page_login.make_login('za2217279@mvrht.net', 'test123')
        page_new_doc = GrammarlyNewDocument(self.driver)
        page_new_doc.make_new_document("")
        page_doc = GrammarlyDocument(self.driver)
        self.assert_text(page_doc)

    def test_change_title_and_text_on_doc(self):
        page_login = GrammarlyLogin(self.driver)
        page_login.make_login('za2217279@mvrht.net', 'test123')
        page_new_doc = GrammarlyNewDocument(self.driver)
        page_new_doc.make_new_document("")
        page_doc = GrammarlyDocument(self.driver)
        self.assert_title(page_doc)
        self.assert_text(page_doc)

    def assert_title(self, page_doc):
        title_expected = "NewTestTitle"
        page_doc.put_title(title_expected)
        time.sleep(2)
        actual_doc = GrammarlyDocument(self.driver)
        self.assertEquals(page_doc.title, actual_doc.title)

    def assert_text(self, page_doc):
        text_expected = "A very bad text that are have some issues and Grammarly Should find problems with it"
        page_doc.put_text(text_expected)
        time.sleep(2)
        actual_doc = GrammarlyDocument(self.driver)
        self.assertEquals(page_doc.text, actual_doc.text)

You can check the whole project in this state at tag: doc-title-and-text

Pelican plugin for Grammarly

That's just a POC for making Grammarly and Selenium Automation. In part2 I'll try to make integration with the pelican engine.

This time I'll focus my attention on gathering as much as I can from the Grammarly document Checker. :)

Acknowledgements

Grammarly Alternatives: - TextGears - this one has an API that you could use. ( and payment for amount of requests/data sent)

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to upgrade python to 3.6.4

2018-02-17T22:25:00+01:00

Today I've learned my lesson - I need to upgrade my python from 3.4.3 to 3.6.4 because while analysing simplenote python-api I've found that it will not work with previous versions. Checkout my experience while upgrading :)

S0-E15/E30 :)

Simplenote Case

Few articles back I've wrote about simplenote and possibility to use it with vim. Unfortunatelly on my Ubuntu-Trusty (14.04) I could not.

That time I've digged the problem a bit - but not as much to find out why it was not working on my machine.

Today I've spend few hours on Docker and validating on which version of python It could work. Finally I've found that problem was with python and that using python 3.6.4 could solve problem with accessing simplenote!

Python upgrade to 3.6.4 from source !

sudo apt-get install -y \
     autotools-dev      \
     blt-dev            \
     bzip2              \
     dpkg-dev           \
     g++-multilib       \
     gcc-multilib       \
     libbluetooth-dev   \
     libbz2-dev         \
     libexpat1-dev      \
     libffi-dev         \
     libffi6            \
     libffi6-dbg        \
     libgdbm-dev        \
     libgpm2            \
     libncursesw5-dev   \
     libreadline-dev    \
     libsqlite3-dev     \
     libssl-dev         \
     libtinfo-dev       \
     mime-support       \
     net-tools          \
     netbase            \
     python-crypto      \
     python-mox3        \
     python-pil         \
     python-ply         \
     quilt              \
     tk-dev             \
     zlib1g-dev

wget https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tgz
tar -zxvf Python-3.6.4.tgz
cd Python-3.6.4/ && ./configure --prefix /usr/local/lib/python3.6.4 --enable-ipv6 && cd -
cd Python-3.6.4/ && make && cd -
cd Python-3.6.4/ && make altinstall

Simplenote problems not resolved :(

Unfortunatelly it looks like upgrading to python3.6.4 is not the only thing I need to do.

After the upgrade, simpenote.py still does not work on vim :/

I will come back to this- but for now I'll just use docker-container with working solution of python3.6.4 with vim 7.4.

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to deploy Docker on DockerHub with Github Push

2018-02-16T22:00:00+01:00

Did you ever wanted to deploy your own docker-image with only git-push ? Well I've got solution for you :) Check-it-out :)

S0-E14/E30 :)

Project in Github

First what you need is a Github repository. Then what you need is to enable docker integration on the project with :

Settings -> Integrations & services -> Add Service -> "Docker"

Docker Hub with access

When you have setup your github repository, what you need to do is go to Docker Hub and create your account if you haven't done it yet. Then go to Create -> Create Automated Build -> Link Accounts Or go to this site.

Then you will link your github account.

After this point you will be able to make automated builds for docker-images with git-pushes.

Docker image on Docker Hub with push

I've created a github-test-repository called docker_discovery that is linked now with github.

Dockerfile :

FROM alpine
MAINTAINER anselmos@users.noreply.github.com
RUN apk update
RUN apk add vim

You can check my first successful build here

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Update your tmux to latest version

2018-02-15T20:40:00+01:00

Today a very short tutorial on how to update tmux at *nix. Why would you update your tmux ? Checkout.

S0-E13/E30 :)

Tmux

Tmux is an terminal multiplexer and a some-how successor of gnu-screen.

I used to use screen, but since it didn't have such good multiplexing as tmux, I've decided to abandon it.

Why do you need latest version of tmux?

That's the problem - since I'm planning to use Ubuntu Trusty disto until 18.04 will be delivered, I need to update my packages from 14.04 manually.

But why ?

There's a plugin called tmux-ressurect that has this fancy option to save & restore all tmux sessions with only CMD + r and CMD + s. Unfortunatelly it needs at least 1.9 version of tmux and trusty disto went with 1.8 :( so let's upgrade to 2.6 :D

Building from source

Let's first install dependencies for building on our machine:

sudo apt-get -y remove tmux
sudo apt-get -y install wget tar libevent-dev libncurses-dev

then let's download source and unpack it:

VERSION=2.6
wget https://github.com/tmux/tmux/releases/download/${VERSION}/tmux-${VERSION}.tar.gz
tar xf tmux-${VERSION}.tar.gz
rm -f tmux-${VERSION}.tar.gz

Now let's build and install it and cleanit up:

VERSION=2.6
cd tmux-${VERSION}
./configure
make
sudo make install
cd -
sudo rm -rf /usr/local/src/tmux-\*
sudo mv tmux-${VERSION} /usr/local/src

Than only thing that you SHOULD do - not to have problems with protocol version mismatch is to kill old tmux server that can be still running in the background with :

sudo killall -9 tmux

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Grip and Vim for Note taking

2018-02-14T19:00:00+01:00

Grip it! There are plenty of tools to make your .markdown or .md files in a html way- but is there a vim plugin that would make it more automated and integrated ? Let's answer this question, shall we ?

S0-E12/E30 :)

Vim note taking and a markdown files.

Vim is my favourite tool for making short notes. Also it works best as a IDE for python files.

But how to make those markdown files available on your machine without having to use any websites or puting it online ?

There's a way - besides those GUI apps that gives you that, you can actually make use of a Grip markdown renderer.

Grip!

Thanks to Joe Esposito you can watch on browser how you type-in your markdown file. Each time you will save your markdown file that is connected with grip, it will automatically refresh.

The project Grip is something that I personally use and used even at work for short exchanging of information within markdown files. You can actually have a server that's on your machine visible within your local network. (putting aside network-alike problems).

But in order to do that you have to have another script run in your system.

How to use it ?

First install grip with a pip (for global installation) or within your environment (first activate your virtual-env)

pip install grip

Than if the file is called README.md grip will automatically make use of it with command grip if you are in the same directory as mentioned file.

Grip actually will make a listing of files and you can access files from directory you started grip command.

I.e. you have 2 files in your directory : README.md and a-very-important-note.md.

Now what grip will do is select the README.md and make a html page when you open your browser at localhost:6419. Also you can go to localhost:6419/a-very-important-node.md and it will render your markdown file into html.

The configuration section within readme of grip gives you even a hint of possibility to change port number on which the service will work.

Vim integrations for markdown files

Is there any type of integration with vim and markdown that you could use ?

Vim-Markdown

It's an integration for vim that reutilize power of vim to make markdown experience more vim-alike.

I.e. this page is a markdown file that I'm currently writing! And I'm using Vim-Markdown that presents a TOC (Table Of Contents) on the right when I put in vim-exec mode :

:Toc

Also you can check all your links in md file if they link to proper site.

Github project site Vim-Markdown by plasticboy.

Vim-Wiki

It's a Markdown integration for making notes as a type of Wiki. You start with your index.md and then you can create all your notes within vim.

Checkout github project page

The only minus of this is that at this point you have to use a special type of syntax and a .wiki extensions.

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Unittests and Assert Raise

2018-02-13T23:50:00+01:00

Using assert raise within tests is a tricky thing - So today I'll focus on how to properly use it :)

S0-E11/E30 :)

Unittest with self.assertRaise for function tests

Let's present example for function:

import unittest

def raiser_function():
    print "Is This function Called?"
    raise NotImplementedError()

class TestCase2(unittest.TestCase):
    def test_will_pass_self_assert_raise(self):
        # proper usage
        self.assertRaises(NotImplementedError, raiser_function)

    def test_with_self_assert_raise(self):
        # proper usage
        with self.assertRaises(NotImplementedError):
            raiser_function()

    def test_will_fail(self):
        # added intentionally this line below for the test not to fail.
        with self.assertRaises(NotImplementedError):
            # you should not call methoc/function inside of self.assertRaises, but give arguments and kwargs as self.assertRaises argument
            self.assertRaises(NotImplementedError, raiser_function())

if __name__ == "__main__":
    unittest.main()

Unittest with self.assertRaise for object method tests

Let's present example for object-method:

import unittest

class RaiserClass(object):
    def methodToRaiseNotImplementedError(self):
        print "Is This Test Called?"
        raise NotImplementedError()

class TestCase1(unittest.TestCase):
    def test_will_pass_self_assert_raise(self):
        self.assertRaises(NotImplementedError, RaiserClass().methodToRaiseNotImplementedError)

    def test_with_self_assert_raise(self):
        with self.assertRaises(NotImplementedError):
            RaiserClass().methodToRaiseNotImplementedError()

    def test_will_fail(self):
        # added intentionally this line below for the test not to fail.
        with self.assertRaises(NotImplementedError):
            raiser_object = RaiserClass()
            self.assertRaises(NotImplementedError, raiser_object.methodToRaiseNotImplementedError())

    def test_will_fail2(self):
        # added intentionally this line below for the test not to fail.
        with self.assertRaises(NotImplementedError):
            self.assertRaises(NotImplementedError, RaiserClass().methodToRaiseNotImplementedError())

if __name__ == "__main__":
    unittest.main()

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Zope Inteface and Its purpose

2018-02-12T18:00:00+01:00

This time let's focus on something different than usually - Let's check what Zope Interface is all about :)

S0-E10/E30 :)

Zope Interface

Zope Interface is a part of so called Zope Toolkit - a set of libraries used within web-frameworks.

This interface is about making a reusable elements in system that you then re-purpose in a DRY principle.

Zope Interface usage

You may define your own interface for class. This may be good for example to know what tests needs to be created:

import unittest
import json
from zope import interface
from zope.interface.verify import verifyClass
from zope.interface.verify import BrokenImplementation
from zope.interface.tests import test_verify


class ITestObject(interface.Interface):
    def test_simplest_leading_zero():
        """ This test will check if LeadingZeroValidator behave as expected"""


class TestObject(object):
    interface.implements(ITestObject)


@interface.implementer(ITestObject)
class TestObjectProperlyImplements(object):
    def test_simplest_leading_zero(self):
        """"""
        raise NotImplementedError()


class TestIfMyTestObjectImplementsMethods(unittest.TestCase):

    def test_verify_fails(self):
        with self.assertRaises(BrokenImplementation) as context:
            self.assertTrue(verifyClass(ITestObject, TestObject))

    def test_verify_success(self):
        verifyClass(ITestObject, TestObjectProperlyImplements)

    def test_using_test_method(self):
        testobj = TestObjectProperlyImplements()
        with self.assertRaises(NotImplementedError):
            testobj.test_simplest_leading_zero()

NotImplementedError vs verifyClass

While I was researching the topic I've found that using anything inside of methods defined in Interface - will be disregarded.

I wanted to create a more abstract Interface that would prompt about NotImplementedError - than I've found that zope.interface.Interface disregards any body inside of the method.

You need to create class based on this interface to be able to use NotImplementedError as in example above.

Below you can check example of this mis-guided behaviour:

import unittest
import json
from zope import interface
from zope.interface.verify import verifyClass
from zope.interface.verify import BrokenImplementation
from zope.interface.tests import test_verify

class IHowNotCreateInterfaceWithMethodBody(interface.Interface):
    def method_that_should_be_implemented():
        """ This method needs to be implemented by object!"""
        # As it's only interface, you should not give body to it, but we want to learn our lesson so:
        # anything that you do in this will be disregarded
        raise NotImplementedError()


class ImplementingHowNotCreateInterface(object):
    interface.implements(IHowNotCreateInterfaceWithMethodBody)

class TestingHowNotCreateInterface(unittest.TestCase):

    def test_verify(self):
        # at this point it will not fail with "NotImplementedError" - instead you can invoke your method and have AttributeError.
        testobj = ImplementingHowNotCreateInterface()
        with self.assertRaises(AttributeError):
            testobj.method_that_should_be_implemented()
        with self.assertRaises(BrokenImplementation):
            verifyClass(IHowNotCreateInterfaceWithMethodBody, ImplementingHowNotCreateInterface)


if __name__ == '__main__':
    unittest.main()

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

Making Own Colander Validator with Tests

2018-02-11T20:05:00+01:00

Today I present you a my own Colander Validators! This includes tests that uses those validators! Check them out!

This episode is a series about colander

S0-E9/E30 :)

Colander Validator Extended

Did you ever needed to create a more advanced validation than what Colander offers? I did!

There is a way to create your own validations for elements. You can even extend pre-existing validations and add your own stuff in it.

Let's say that you want to make sure at the validation step that data you will pass-on will have leading zero in one field.

There are no validators like that in standard Colander library - you would need to extend it.

Ofcourse you could use Regexp also.

Let's make that, shall we ? :)

import json
import colander


class LeadingZeroValidator(object):
    """ Validator for specific check if value has leading 0"""

    def __call__(self, node, value):
        if str(value)[0] != "0":
            raise colander.Invalid(node, "No leading 0: %s"%(value))

class User(colander.MappingSchema):
    id = colander.SchemaNode(colander.String(), validator=LeadingZeroValidator())
    name = colander.SchemaNode(colander.String())


class Users(colander.SequenceSchema):
    user = User()


class Data(colander.MappingSchema):
    users = Users()
    list_name = colander.SchemaNode(colander.String())


def get_example_data():
    return {
        "users": [
            {
                "id": "2",
                "name": "Anselmos",
            },
            {
                "id": "2",
                "name": "Somlesna",
            }
        ],
        "list_name": "name_of_users"
    }


def get_validation_data(dict_data):

    api_data = json.dumps(dict_data)
    # print api_data
    json_loaded = json.loads(api_data)
    dicted = dict(json_loaded)
    return dicted


def make_validation(data):

    serialized = Data().serialize(data)
    deserialized = Data().deserialize(data)


if __name__ == '__main__':
    data = get_validation_data(get_example_data())
    make_validation(data)

This code will now give information at the make_validation step (Data().deserialize(data)), that there is element with no leading zero value.

Using Colander in Unittests

Now let's make a unittest that will validate if our LeadingZeroValidator is working properly:

import unittest
import json
import colander

from extended_validation1 import get_validation_data, make_validation
from extended_validation1 import LeadingZeroValidator

class TestLeadingZeroValidator(unittest.TestCase):

    def test_simplest_leading_zero(self):
        test_case = "01"
        validator = LeadingZeroValidator()
        validator("", test_case)

    def test_simplest_failing_leading_zero(self):
        test_case = "1"
        validator = LeadingZeroValidator()
        with self.assertRaises(colander.Invalid) as context:
            validator("", test_case)

    def test_invalid_assert_raises(self):
        dict_1 = {
            "users": [
                {
                    "id": "2",
                    "name": "Anselmos",
                },
                {
                    "id": "02",
                    "name": "Somlesna",
                }
            ],
            "list_name": "name_of_users"
        }

        testcase1 = get_validation_data(dict_1)
        with self.assertRaises(colander.Invalid) as context:
            make_validation(testcase1)

    def test_invalid_not_raises(self):
        dict_1 = {
            "users": [
                {
                    "id": "01",
                    "name": "2Anselmos",
                },
                {
                    "id": "02",
                    "name": "2Somlesna",
                }
            ],
            "list_name": "name_of_users"
        }

        testcase1 = get_validation_data(dict_1)
        make_validation(testcase1)

if __name__ == '__main__':
    unittest.main()

And it does :) how sweet.

Where to go from here ?

You may check also information about SchemaTypes and creating your own SchemaType! :)

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

BTW - today is the last day when I eat junk food - why ? Because I'm starting another challenge for not eating junk food for the time of fasting - so at least 40 days :) - So Keep finger crossed for me :)

See you tomorrow! Cheers!

Using Colander Validators with nested elements

2018-02-10T23:10:00+01:00

In todays episode we will use Colander with validators for nested elements :) Checkout my experience with them.

This episode is a series about colander

S0-E8/E30 :)

Colander Nested

Let's say you have a nested list elements that also have atributes like this :

{
    "users": [
        {
            "id": "1",
            "name": "Anselmos",
        },
        {
            "id": "1",
            "name": "Somlesna",
        }
    ],
    "list_name": "name_of_users"
}

Code that I would typically create would look like this:

import json
dict_1 = {
    "users": [
        {
            "id": "1",
            "name": "Anselmos",
        },
        {
            "id": "1",
            "name": "Somlesna",
        }
    ],
    "list_name": "name_of_users"
}
api_data = json.dumps(dict_1)
print api_data
json_loaded = json.loads(api_data)
dicted = dict(json_loaded)
print dicted

class JsonDeserialize:
    def __init__(self, **entry):
        self.__dict__.update(entry)

classed = JsonDeserialize(**dicted)
print classed.users
print classed.users[0]

What is the problem with it ? Well you still have to manually create a validator for each user list-element.

Colander can fix that with:

import json
import colander
dict_1 = {
    "users": [
        {
            "id": "1",
            "name": "Anselmos",
        },
        {
            "id": "1",
            "name": "Somlesna",
        }
    ],
    "list_name": "name_of_users"
}
api_data = json.dumps(dict_1)
print api_data
json_loaded = json.loads(api_data)
dicted = dict(json_loaded)
print dicted

class User(colander.MappingSchema):
    id = colander.SchemaNode(colander.Int())
    name = colander.SchemaNode(colander.String())

class Users(colander.SequenceSchema):
    user = User()

class Data(colander.MappingSchema):
    users = Users()
    list_name = colander.SchemaNode(colander.String())

serialized= Data().serialize(dicted)
import pprint
print pprint.pformat(serialized)
deserialized = Data().deserialize(dicted)
print pprint.pformat(deserialized)

Now I now this looks pretty simple. But let's make another example that will use validator - which you could not create with simple json and dict "hack" :

import json
import colander
dict_1 = {
    "users": [
        {
            "id": "1",
            "name": "Anselmos",
        },
        {
            "id": "1",
            "name": "Somlesna",
        }
    ],
    "list_name": "name_of_users"
}
api_data = json.dumps(dict_1)
print api_data
json_loaded = json.loads(api_data)
dicted = dict(json_loaded)
print dicted

class User(colander.MappingSchema):
    id = colander.SchemaNode(colander.Int(), validator=colander.NoneOf([1]))
    name = colander.SchemaNode(colander.String())

class Users(colander.SequenceSchema):
    user = User()

class Data(colander.MappingSchema):
    users = Users()
    list_name = colander.SchemaNode(colander.String())

serialized= Data().serialize(dicted)
import pprint
print pprint.pformat(serialized)
deserialized = Data().deserialize(dicted)
print pprint.pformat(deserialized)

Oh ... now it will fail with :

local/lib/python2.7/site-packages/colander/__init__.py", line 704, in _impl
    raise error
colander.Invalid: {'users.0.id': u'"1" must not be one of 1',
 'users.1.id': u'"1" must not be one of 1'}

Because we added to our schema validator like that:

    id = colander.SchemaNode(colander.Int(), validator=colander.NoneOf([1]))

Ofcourse the obvious fix for that will be to change id's for user's to different than 1.

Colander build-in validators

There are 13 build-in colander validators that we can make great use.

I encourage you to try them :)

1. colander.All(*validators)

2. colander.Any(*validators)

3. colander.Range(min=None, max=None, min_err='${val} is less than minimum value ${min}', max_err='${val} is greater than maximum value ${max}')

4. colander.Length(min=None, max=None, min_err='Shorter than minimum length ${min}', max_err='Longer than maximum length ${max}')

5. colander.OneOf(choices)

6. colander.NoneOf(choices, msg_err='"${val}" must not be one of ${choices}')

7. colander.ContainsOnly(choices)

8. colander.Function(function, msg=None, message=None)

9. colander.Regex(regex, msg=None, flags=0)

10. colander.Email(msg=None)

11. colander.luhnok(node, value)

12. colander.url

13. colander.uuid

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to use Colander - part1

2018-02-09T23:00:00+01:00

Did you ever wonder what are alternatives for serializing and deserializing besides of simple json de/serializer ? I didn't but I've found this library that I find pretty neat :)

What Colander is all about ?

Well, to be honest I would not found this library if it was not for pyramid!

This library is a part of pyramid that integrates well into this pyramid ;D (yeah a bit to dry humour :D)

This library's main purpose is a type of serialization/deserialization mechanism with validation that can be extended.

How to use Colander?

You can use that i.e. within your small scripts / small frameworks that does not have a predefined serializers or where those serializers does not give you any type of extending for validating data.

In frameworks like django, you should find colander not much usefull, but in flask - I would take it - unless I'd find something more advanced at those frameworks.

So how to use it ? Let's have a look at small script that I've created for the sake of this:

import json

json_dict = {
    'height': 175,
    'surname': 'Somlesna',
    'id': 'None',
    'weight': 80,
    'name': 'Anselmos',
    'bmi': 26.12,
    'bmi_health_name': 'Overweight'
}
api_json_request_simulated = json.dumps(json_dict)

class JsonDeserialize:
    def __init__(self, **entry):
        self.__dict__.update(entry)

dict_deserialized = json.loads(api_json_request_simulated)
test1 = dict(dict_deserialized)
# no walidation at this point! in colander you have already a validation at this point! that would prevent having something that should not be given validated.

print JsonDeserialize(**test1)
print JsonDeserialize(**test1).bmi
print JsonDeserialize(**test1).bmi_health_name

This is a type of serialization to object that you could use. This ofcourse works. Main problem with it is that for making a validation you would need to replace mechanism for creating object from dict.

There are no way for json.loads/dumps to validate data (at least I don't know about).

But if someone from the blog audiance know about, please let me know :)

With colander you can validate your data before decoding it to dict:

import json
import logging
import pprint
import colander

json_dict = {
    'height': 175,
    'surname': 'Somlesna',
    'id': 'None',
    'weight': 80,
    'name': 'Anselmos',
    'bmi': 26.12,
    'bmi_health_name': 'Overweight'
}
api_json_request_simulated = json.dumps(json_dict)
dict_deserialized = json.loads(api_json_request_simulated)
test1 = dict(dict_deserialized)

class User(colander.MappingSchema):
    id = colander.SchemaNode(colander.Int())
    height = colander.SchemaNode(colander.Int(), validator=colander.Range(0,400))
    surname = colander.SchemaNode(colander.String())
    weight = colander.SchemaNode(colander.Int())
    name = colander.SchemaNode(colander.String())
    bmi = colander.SchemaNode(colander.Int())
    bmi_health_name = colander.SchemaNode(colander.String())

deserialized_obj = User().serialize(dict_deserialized)

This code will give you error! Why? because main int-validator for ID has triggered:

/local/lib/python2.7/site-packages/colander/__init__.py", line 704, in _impl
    raise error
    colander.Invalid: {'id': u'"None" is not a number'}

After a bit of changing now second validator will occur in this example:

import json
import logging
import pprint
import colander

json_dict = {
    'height': 402,
    'surname': 'Somlesna',
    'id': 1,
    'weight': 80,
    'name': 'Anselmos',
    'bmi': 26.12,
    'bmi_health_name': 'Overweight'
}
api_json_request_simulated = json.dumps(json_dict)
dict_deserialized = json.loads(api_json_request_simulated)
test1 = dict(dict_deserialized)

class User(colander.MappingSchema):
    id = colander.SchemaNode(colander.Int())
    height = colander.SchemaNode(colander.Int(), validator=colander.Range(0,400))
    surname = colander.SchemaNode(colander.String())
    weight = colander.SchemaNode(colander.Int())
    name = colander.SchemaNode(colander.String())
    bmi = colander.SchemaNode(colander.Int())
    bmi_health_name = colander.SchemaNode(colander.String())

deserialized_obj = User().deserialize(dict_deserialized)

Will output:

local/lib/python2.7/site-packages/colander/__init__.py", line 704, in _impl
    raise error
    colander.Invalid: {'height': u'402 is greater than maximum value 400'}

Colander schemas and types.

This rather small tutorial does not end on this. Colander has many schema-types and extensions that I want do share with you in next episode! So stay tuned!

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to integrate vim with simplenote? Is there a way for that?

2018-02-08T07:00:00+01:00

S0-E6/E30 :)

How to integrate your SimpleNote account and your vim editor ?

There are a simple tutorial how to make your SimpleNote integrated with VIM.

Use mrtazz simplenote.vim repo - as follows:

Add this to your .vimrc file:

Plugin 'mrtazz/simplenote.vim'

let g:SimplenoteUsername = "your-email-here"
let g:SimplenotePassword = "your-password-here"

And then after re-activating your shell env, you should be able to use vim +SimplenoteList - or just make vim and then in vim press: :SimplenoteList to list all your Simplenote Lists.

Failure to connect to server ?

After a bit of digging and testing I've unfortunatelly found that simplenote.py does not authorize user (it was few months ago when I've first found this project), not creating a proper token data- thous making impossible to use this plugin.

I have approached mrtazz on twitter which you can find here - but in the meantime I will try to investigate this problem and find a solution - then make a github PR to his repo :)

What's the purpose of this?

Why would you need a vim plugin to a SimpleNote ? Well - for example if you want to make some type of integration with Grammarly.

You could make your notes within SimpleNote by saving it with vim and then just go to browser and use Grammarly plugins in browser to fix any issues with your note/article/doc.

Final result

This article does not end on this un-happy scenario. I will create another with updates. There will be a Part2 of your SimpleNote-Vim Adventure :)

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to fix removing docker.io on ubuntu

2018-02-07T21:30:00+01:00

S0-E5/E30 :)

How to fix problem on `shopt` with `nuke-graph-directory.sh`

Did you have a problem while sudo apt-get purge docker.io ?

Was this problem related some-how with a shopt like this:

./contrib/nuke-graph-directory.sh: 64: ./contrib/nuke-graph-directory.sh: shopt: not found

I have a fix for you - the fix is so trivial that untill I've found this issue on github I did not realize that this nuke-graph-directory.sh could run on different shell environment!

To fix problem you have to change shebang for this script:

/var/lib/docker/nuke-graph-directory.sh

With what tianon have commited in this commit nuke-graph-directory.sh file

So instead of : -#!/bin/sh - change to #!/usr/bin/env bash

Why do I write about it today ?

Well... that's my little secret I have to confess - I did not checked yesterdays commands from article at the time of creating it.

I was almost 100% sure command apt-get purge will work.

Guess what - when I've check it today on my computer it failed :( That's why I decided to write about this so everyone that will find this issue will know what to do :)

Final result

You should have a fully remove docker.io from your system.

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to upgrade docker engine to latest distribution without breaking any containers

2018-02-06T20:20:00+01:00

S0-E4/E30 :)

Prerequisites

First check from which repository you have installed your docker.

In my case it was docker.io with apt-get install docker.io.

Check with command:

sudo apt list | egrep -i "docker" | egrep -i "installed"

This will list you all packages you have installed named "docker" and that have "installed" information. Something like this:

WARNING: apt does not have a stable CLI interface yet. Use with caution in scripts.

docker/trusty-updates,now 1.5-1 amd64 [installed]
docker.io/trusty-updates,now 1.6.2~dfsg1-1ubuntu4~14.04.1 amd64 [installed]

BTW - you can ignore the warning for this command - because we only checked which packages are installed, not performing actual installing via apt.

If you see something different then docker-ce or docker-ee - then it means you have an older version like docker.io / docker-engine.

That means - You classified to be upgraded ! :)

Manual upgrade with aptitude.

First make update on packages:

sudo apt-get update

Then purge packages that are old:

sudo apt-get purge -y docker.io docker-engine

Then install latest docker (EE - Enterprice Edition, CE- Community Edition) : docker-ce /docker-ee: ! WARNING - please be careful - you should change the "ubuntu-trusty" to your distribution name.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-add-repository 'deb https://apt.dockerproject.org/repo ubuntu-trusty main'
sudo apt-get update
sudo apt-get install -y docker-ce

Upgrade using docker-machine

If you have used docker-machine and created a Virtual Host, than you can just type:

docker-machine upgrade default

where "default" is the name of your machine config.

BTW - you actually can use docker-machine to manage an physical machine with docker installed Check this issue on docker-machine repository.

Final result

You should have a fully working docker with latest engine and now be able to use commands like i.e. "docker network"

Acknowledgements

Thanks!

That's it :) Comment, share or don't :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to create your Selenium Tests with PhantomJS browser integration

2018-02-05T23:50:00+01:00

S0-E3/E30 :)

Did you created your selenium grid tests ? Did you know, that you can get them to run faster? Just use PhantomJS! Wait... what? NodeJS technology in Python stack?! YEAH! With docker you can combine them!

PhantomJS.

PhantomJS is a Javascript Headless browser that at least theoretically should run faster then Firefox.

This article is connected with How-To-Create-Your-Selenium-Grid-Compatible-Test - check it out :)

How to run your PhantomJS

Using docker you can just:

docker run -d -p 8910:8910 wernight/phantomjs phantomjs --webdriver=8910

Your testcase.

And here is what I thought will be a simple integration with Selenium Grid. Then I've found it does not have proper connector to Selenium Grid (eh...). Moreover I've found it has problems with making "send_keys" because of some strange dependency problems that you can find on stackoverflow (problem exists I've check it, before I found those SO pages): - here - here

So what can we do?

Well here's what I've been able to make with a not-fully working python api to remote PhantomJS :

import unittest
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

import time

class PhantomJSTests(unittest.TestCase):
    def test_search_in_python_org(self):
        # this test will not work on PhantomJS since it has a in-compatible API :(
        # instead we need to create a simulation for that
        driver = self.driver
        print driver.get("https://github.com")
        assert "GitHub" in driver.title
        elem = driver.find_element_by_name("q")
        # send_keys does not work on phantomjs - there are some strange workarounds but in my limited time I could not get them to work.
        # I will however make a follow up on this!
        # elem.send_keys("anselmos")
        # time.sleep(5)
        # elem.send_keys(Keys.ENTER)
        # time.sleep(5)
        # assert "No results found." not in driver.page_source

    def setUp(self):
        self.driver = webdriver.Remote(
            command_executor='http://127.0.0.1:8910',
            desired_capabilities=DesiredCapabilities.PHANTOMJS
        )

if __name__ == "__main__":
    unittest.main()

OR download it from here

Is there any other solutions to this ?

Well.... yeah. You can either recompile PhantomJS (by using different branch named " backport-ghostdriver-2-0-to-phantomjs-2-1 " and then using command docker run -v $PWD:/src debian:wheezy /src/deploy/docker-build.sh as suggested on github issue page in this comment by jesg

Wait until someone will fix PhantomJS (that is apparently untill 2.5 -btw current ver. is 2.1.3 ... )

Or... use a Javascript API instead as described here

Thanks!

That's it :) Comment, share or don't :)

I hope you enjoyed this episode as much as I did :)

BTW I will do a follow up on problems with PhantomJS - and will give you feedback if there are any other solutions to the problem :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to create your first Selenium Grid Compatible Test

2018-02-04T17:10:00+01:00

S0-E2/E30 :)

How do you start your first Selenium Grid Test? Yesterday I've given you instructions how to create your Selenium Grid environment. Today I'll focus on how to create your first test that will use this env. Check it Out!

Selenium Grid.

This article is connected with How-To-Start-With-Selenium-Grid - check it out how to create your Selenium Grid Environment :)

Your First testcase.

What to start with? You can start with your own Selenium Test and change it into Selenium Grid one :)

Let's check this example:

import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

class PythonOrgSearch(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Firefox()

    def test_search_in_python_org(self):
        driver = self.driver
        driver.get("https://github.com")
        assert "GitHub" in driver.title
        elem = driver.find_element_by_name("q")
        elem.send_keys("anselmos")
        elem.send_keys(Keys.RETURN)
        assert "No results found." not in driver.page_source

    def tearDown(self):
        self.driver.close()

if __name__ == "__main__":
    unittest.main()

OR download it from here

While I tried to run this on my machine, I've used pip to install selenium, then I've found a problem :

.env/local/lib/python2.7/site-packages/selenium/webdriver/common/service.py", line 83, in start
    os.path.basename(self.path), self.start_error_message)
    WebDriverException: Message: 'geckodriver' executable needs to be in PATH.

To fix this issue you have to :

1. Download a geckodriver from here

As I'm using an x64 bit Linux, I chose geckodriver-v0.19.1-linux64.tar.gz and downloaded it with

wget https://github.com/mozilla/geckodriver/releases/download/v0.19.1/geckodriver-v0.19.1-linux64.tar.gz

Then ungzip it:

tar -zxvf geckodriver-v0.19.1-linux64.tar.gz

And moved into path which is executable: (Warning - you'll need root's permissions to move to /usr path)

mv geckodriver /usr/local/bin/

And then I could finally run the script with :

python your-non-selenium-grid-test.py

Now to make this Test run on our Selenium Grid, just change the "driver" in script with:

self.driver = webdriver.Remote(
    command_executor='http://127.0.0.1:4444/wd/hub',
    desired_capabilities={'browserName': 'firefox', 'javascriptEnabled': True}
)

Final Result - A working example of Selenium Grid Test

import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

class PythonOrgSearch(unittest.TestCase):

    def setUp(self):
        self.driver = webdriver.Remote(
            command_executor='http://127.0.0.1:4444/wd/hub',
            desired_capabilities={'browserName': 'firefox', 'javascriptEnabled': True}
        )

    def test_search_in_python_org(self):
        driver = self.driver
        driver.get("https://github.com")
        assert "GitHub" in driver.title
        elem = driver.find_element_by_name("q")
        elem.send_keys("anselmos")
        elem.send_keys(Keys.RETURN)
        assert "No results found." not in driver.page_source

    def tearDown(self):
        self.driver.close()

if __name__ == "__main__":
    unittest.main()

You can download it also from here.

Execution on more than one node!

If you need to have more than one selenium-node type (besides Firefox), you can customize the script with the following:

First, create a generalized Unittest that will have only test-cases.

import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

class GenericSeleniumTest(unittest.TestCase):

    def setUp(self):
        self.driver = False

    def test_search_in_python_org(self):
        if not self.driver:
            return True
        driver = self.driver
        driver.get("https://github.com")
        assert "GitHub" in driver.title
        elem = driver.find_element_by_name("q")
        elem.send_keys("anselmos")
        elem.send_keys(Keys.RETURN)
        assert "No results found." not in driver.page_source

    def tearDown(self):
        if self.driver:
            self.driver.close()

class FirefoxTest(GenericSeleniumTest):
    def setUp(self):
        self.driver = webdriver.Remote(
            command_executor='http://127.0.0.1:4444/wd/hub',
            desired_capabilities={'browserName': 'firefox', 'javascriptEnabled': True}
        )

class ChromeTests(GenericSeleniumTest):
    def setUp(self):
        self.driver = webdriver.Remote(
            command_executor='http://127.0.0.1:4444/wd/hub',
            desired_capabilities={'browserName': 'chrome', 'javascriptEnabled': True}
        )

if __name__ == "__main__":
    unittest.main()

And that's it! You have a fully working and customizable Tests. You can now extend Generalized tests that will be run on each platform or perform a specific test designed only for Firefox or Chrome.

Acknowledgement

Thanks!

That's it :) Comment, share or don't :)

I hope you enjoyed this episode as much as I did :)

BTW: I have something extra for you - since I've used yesterdays commands, I created a simpler script for non-network compatible docker-engines that creates a selenium grid out of the box Check it Here (Only for security reasons I'm putting Md5Sum of this file: 7531d8a83d2f9344a752b203e8e1198c run_selenium_grid.sh ) :)

If you have any suggestions what I should blog about in the next articles - please give me a hint :)

See you tomorrow! Cheers!

How to start with Selenium Grid ? Step by Step.

2018-02-03T22:30:00+01:00

Day 1 out of 30 :)

How do you start with selenium grid? From scratch ? When you have no idea of what that is ? Today I'm going to give you answers to that questions! Check it out and share if you liked it !

What Selenium Grid is, and what is not ?

For sure is not a silver lining for automation. And for sure Selenium Grid for beginner is not a simple solution.

That's why I create this blog post - for others to start using it with simpler step-by-step guides :)

Selenium Grid is a type of server, that you can use for making Selenium Automation Testing without having a GUI - so for typical server-type computer.

Selenium Grid has it's potential to make a better testing environment - because you can plug-and-play using docker and then plug it to some type of CI.

In that way - you can fully combine your web-automation testing into your CI environment.

What Selenium Grid is made of ?

Selenium Grid consists of Hub and Nodes.

Hub is the Central point to which you send commands about your tests.

Nodes are just regular workers - you push your tests via Hub to them. Nodes can be a of a different type - for example a Firefox Browser Node, or a Mozilla Google Browser Node.

How to start Selenium Grid?

For start, install docker Installation Tips from DigitalOcean for Ubuntu 16.04

Then It's pretty easy to make a Selenium Hub with Nodes. Just follow this instructions Here is link to gh-source for documentation:

Create a docker-internal network

Creating for nodes and hub to easier recognize themselves via network:

docker network create grid

Warning - if you have problems with docker network command

You should check whether your docker-engine version is of 1.22 API - how ?

docker version Example :

Server API version: 1.18

Mine unfortunatelly has a 1.18 :( so I'm stuck with no-docker-network)

You can also check Docker Engine Versions.

Create a Hub (including docker network)

docker run -d -p 4444:4444 --net grid --name selenium-hub selenium/hub:3.8.1-francium

Create a Hub ( without docker network):

docker run -d -p 4444:4444 --name selenium-hub selenium/hub:3.8.1-francium

Now you can also check whether a hub has started it's job, by checking:

docker logs selenium-hub

If you see something like :

- Selenium Grid hub is up and running as a last line from log - that it means you have successfully created a working Selenium Hub ! :)

Create Nodes (including docker network)

docker run -d --net grid -e HUB_HOST=selenium-hub -v /dev/shm:/dev/shm selenium/node-chrome:3.8.1-francium
docker run -d --net grid -e HUB_HOST=selenium-hub -v /dev/shm:/dev/shm selenium/node-firefox:3.8.1-francium

Create Nodes (without docker network)

docker run -d --link selenium-hub:hub --name selenium-node-chrome -v /dev/shm:/dev/shm selenium/node-chrome:3.8.1-francium
docker run -d --link selenium-hub:hub --name selenium-node-firefox -v /dev/shm:/dev/shm selenium/node-firefox:3.8.1-francium

If you can see from:

docker logs selenium-node-chrome

Something like this:

22:11:06.319 INFO - Selenium Grid node is up and ready to register to the hub
22:11:06.324 INFO - Starting auto registration thread. Will try to register every 5000 ms.
22:11:06.325 INFO - Registering the node to the hub: http://172.17.0.2:4444/grid/register
22:11:06.377 INFO - The node is registered to the hub and ready to use

Than it means you have successfully registered a Selenium Node ! Which means you have successfully created a Selenium Grid Stack from which you can create your automated tests within CI :)

Thanks!

That's it :) Comment, share or don't :)

BTW : I have a Todoist redeem code for anyone how will give me a good idea for next article :)

BTW! This post is inspired by MiroBurn VLog S3E34 ! Totally out of the blue I've decided to accept his challenge of 30-days contributing to society(in my version it's a 30 days of posting an article for blog).

Keep Finger crossed for me :)

See you tomorrow! Cheers!

How to start New 2018 Year?

2018-01-11T07:00:00+01:00

Before we begin with how to start new year, let me summarize 2017:

Summarizing 2017

It's been a long journey for me to get back to blogging. Since the last article was published, I have found billions of excuses not to blog, and other excuses not to make any progress with my BikingEndorphines project :(

As the result of that, I did not complete project in a beta-phase.

That's about failures.... - there was more failures, but is there any sunshine out-there?

Yes! I'm writing this article! That's a huge step to get back with my unfinished businesses.

What else ... Let's summarize 2017 in 10 brief points:

1. Challenged myself to create this blog and post in a DSP competition articles about BikingEndorphines project

2. Finally broke the ice with flying tourist jets.

3. Came back to driving a car - something that I did not anticipate myself that would be possible.

4. Started Master-Mind sessions with my 2 friends - Special thanks for @Wojciech

5. Went to conferences- which embraced my ambitions.

6. Came-back to drawing on paper

7. Decided it was time to change my employer.

8. Started early morning routines (waking up at 5 am or earlier).

9. Successfully completed Associate Android Developer course by Udacity, certified by Google (https://developers.google.com/training/certification/associate-android-developer/)

10. Finally decided to buy a better phone to be more mobile.

So... How to start New Year?

How about from... Happy New Year! :)

But ehm.... seriously ?

Everyone has made their commitments for this year. If you did not, you are the lucky sailor above this challenges.

I also made my commitments. This time I've finally used "3 layers" or "3 Poziomy" (in polish lang) to keep my goals real and more achievable ( you can check out about "3 poziomy" https://trzypoziomy.pl/zadania/ ).

But how can you change your new year and be more of yourself in this year?

Let's ask yourself a few questions that may lead you to the path you were looking for, shall we?

First of all, let's start with a big one - where do you see yourself in 5 years?

That's the question I've been asked year by year on a "yearly work employee reviews" - of course, it had work-related context, but let's think about this in a bit more comprehensive way.

Let's for example assume that in past you have wonder how it would be to become a great piano artist. Maybe you thought about taking that piano lessons but since you think there is no time for that, you did not even started?

For this example, let's remove money from the equation. Let's focus only on time needed to master in a beginning level piano playing.

Learning how to play a piano needs - that's my imagination - 100hours.

If you are working/walking to school and sleeping regularly, it takes about 16 hours in average. Add time needed for transport and preparation to work/school and you are left with about 4-5 hours of free time that you can spend on anything you want.

Sky is the limit, right ?

Yes and no. The only one that divides you from winning is yourself and your motivations. But let's come back to our calculations.

Let's assume that you want to spend at least 1 hour a day for relax - i.e. playing games/reading books/watching movies/tv-series. That means you have about 4 hours of free time per work day that you could spend on something productive and/or for expending your relations with others.

If you want to spend 1 hour per day for learning how to play piano, you would need :

100h / 5 hours per week = 20 weeks = approx. 5 months.

That's less then half year! (Remember - half-full glass of water )

Think, what you could do with second half of the year ? :-)

My 2018 plans?

I you wonder what are my plans(especially connected with blog-context):

1. Blogging at least twice per month

2. Focus on my BikingEndorphines project

3. Change deployment system of new articles to blog into more mobile way

Thanks!

That's it :) Comment and share if you liked this article.

BTW : I'd appreciate any suggestions about next articles.

PS Next time there will be a more technical stuff :) but Ciiiiiii - no-one needs to know :-)

See you! Cheers :)

How to use Stack Hive?

2017-08-16T21:30:00+02:00

--EDIT 28-02-2018 Stackhive Deprecated!

From 1.03.2018 StackHive is no longer available. Informations below does not apply anymore :(

About StackHive

In the last article, we talked about how to use some online-designing tools that could help out-of-the-box to design your own website/product.

I've picked StackHive to check it's abilities to create a dynamic web page with a source-code visible with a "WYSIWYG" editor-type.

Now I have one warning for those not familiar with this product - You will probably need to wait at least few days (in my case it was about 3 days) till my account was fully activated and I could use their interface to make changes in projects.

By default account you create will be a free-type and you can use it for creating max 3 projects with the ability to edition and exporting.

There may be other repercussions (that some features may not work in the free-account type) but I do hope that most of the features I'll be using will work :)

Enough chit-chat - let's go with our initial project.

First project in StackHive.

After making an account, you can "Create your project" by clicking one of 3 options : - "Start a project with our templates" - "Upload your own code via zip" - "Pull from live website"

For now, I only used the first option. Just for this post I"ve started the third option to check if it will download my blog website for the edition.

Choosing project from templates

You can use free-templates within free-account. There are (at least I suppose) non-free templates that may be more advanced - out of the box.

Let's assume you picked a "Blog Template" or any other template.

Now at the "Manage Projects" section, you will find your projects that either you imported or created from a template.

I used my 3 free projects, the first project is for testing importing my blog for checking the "Pull from live website" - I must admit it works pretty great for a static-generated blog.

Other 2 projects are my new-template based projects.

Going back to the business - you can either "View Project", "Edit Project" or click the "trash-bin" icon to remove the project, once you move the cursor on the project.

Let's see what editor can do for us :)

StackHive Editor!

As you may see now, the editor has a cool option. You have the ability to "Preview" your "work-in-progress" site, to check how it will look on a different type of devices i.e. Tablet/ Phone/ HD Desktop. It also includes the Portrait and Landscape mode.

Also, there is the "Plus" sign with "Create" name, that gives the possibility to add new elements to the website.

This element comes with filtering, just as in other graphical editors.

I particularly searched for "list" and 3 elements returned "List", "ListItem" and "Navigation List"

Images and uploading

In this fantastic editor, you can change and upload your own images that can be put anywhere in the project.

Publishing the website (staging/production)

A great thing that you can also check is an option to publish your own live version of the site you were working on.

Just click at Manager (at the left site) and then the "Stack" icon. Then you will be able to choose from a subdomain of "stackhive.com" for free or at your own custom domain (that you paid).

WOW - that was my impression after using it! After making some changes in the theme, I've gone to the "Publish" options and clicked "Push" - it immediately pushed changes I've made into the subdomain.

You may check my initial and very drafty version of the site here:

BikingEndorphines-On-StackHive

BTW - Great support from StackHive service technicians!

At the middle of writing of this article, I could not use the service, because of some domain problem.

Fortunately, after reaching out to StackHive on Facebook, after only an hour, the problem was solved - thanks again!

Comment and share :)

Afterall (I've changed the publish date few times) - StackHive looks like a very intuitive site for building sites! I will definitely use it in the future.

I'd love to hear an opinion from you! Would you use StackHive?

What other online-design tools you use?

Thanks and see you soon!

Designing Biking Endorphines - what went wrong?

2017-08-04T20:00:00+02:00

Original Design

Initially I did not have any design at all. I just planned my work few steps ahead for development.

I didn't realize untill lately, that I did a terrible thing - I didn't have an idea of how "almost finished" product will look like with Biking Endorphines project.

Fortunatelly for me, my colleagues from our "MasterMind" group gave me a great feedback about that - Thanks @Grzegorz and @Wojtek.

I gave too much effort and was focused almost 100% on the code instead of the product itself.

That's why I've decided to cross-out all the plans I had and start almost from begining with designing web-based application.

In that situation I will know where the work needs to go (the path I'm leading it) and know how much there is left to do.

And then I can focus on the REST-API and Android APP. (which btw will also need to be designed first)

Web Design tools/services

A part of me didn't want to do this in an online-tool. Instead I'd prefer to make a drawing by hand sketch.

But online and the "software" tools have a lot of things already prepared for job I want to do that it would not be productive - it would be a bit of wasting time on sketching.

I will probably do some hand-sketching after-all just as a POF- but for now let's focus on what the Web has for us.

Tools & Services I've found for making web design:

This is a recurring one from Tim Ferris - "99designs.com"
The Bootstrap Studio - not free - when I've checkout it out- it was about 25$ for a "lifetime" offline licence.
StackHive - online web-based application for making designs of web-application. What is great about this app- it has a "free" plan in which you have 2 projects available. Also this application allows you to export your project and have it used elsewhere.
Tools like Weebly, Webs.com - where you probably don't have access to files you created - platform provides included hosting of web-site created in online-creator
Much more, checkout : AlternativeToLink Of BootstrapStudio

B-Endorphines Web Design

Well it's a tough decision, but for now I'll probably stick with the "StackHive" - to checkout they're free-plan. If this will not work-out as I would want to, I might even think about buying the "Bootstrap Studio" - based on their demo it looks also promising.

Biking Endorphines design will contain elements such as:

1. Login Page

2. Demo site page where the diagrams and functionality is presented.

3. Biking Statistics page sorted by day/week/month/year and even by route-location-points, that you could filter.

4. Biking Endorphines Badges Acquired page that will contain all the Badges user gathered (when logged-in) and information which badges are yet to be discovered (some of them will be hidden).

And so I've created account on "StackHive" and choosed template called "Pratt Theme".

It looks like after creating a new project in "StackHive" (from template) it needs to be loaded within some type of a Queue as the message follows: "Please wait while we load your website".

As I'm currently in a bus using my 3G internet - if the "loading" is not on the server site, but on the client site, it will take a bit longer than I thought.

Well it looks like for the final thoughts about "StackHive" you will need to wait till next article.

Making commitment to stage project into steps

That's the next thing I need to do. I need to stage all the steps and user-stories into plan and then start by making these steps as easy to achive as possible.

First "chunk" of story was to create this article with information about designing B-Endorophines project.

Then I will need to re-think if using Github Issues was such a great idea.

I might even think about moving into some other platform after-all. We will see.

I will for sure make list of the features splitted into as small "chunks" as possible and setup this as my tasks per day/week/month.

Thanks for reading!

Thanks for reading! Please stay tuned - I will make next article about using "StackHive" :)

Please share and comment in the box bellow :) Till next time!

Web Clicking Automation using Selenium IDE test recording

2017-07-27T22:20:00+02:00

Selenium IDE for those in hurry.

Have you got into a situation where you had to test your latest development feature manually, not using TDD?

Maybe you want to have an automation to make your tedious work for you?

In both situations, Selenium IDE may be your friend :)

IDE Test recording

Selenium has an IDE add-on that is fully integrated with Firefox Find it HERE. Originally it was designed for making a simplified test-case recording if you don't need to create more advanced stuff using programming languages (i.e. Page Object Models ).

After you install the addon, it almost looks like nothing happen. But yet you have already installed IDE - it's available at Open Menu (those three dashes) then Developer and then you click Selenium IDE - it should be the first element in Developer context menu.

Then you can see this type of box:

Ther you can see the red ball that is for recording. When you start it and go to the page from Base URL and make some action in there, it will start to be recorded.

How to use Selenium IDE (record your first automation).

Setup Base URL - the URL for which your automation should happen.
Start recording -Just click the red ball at the right top side.
Do on the site whatever you need to/want to automate.
When you are done with the task that you wanted to automate - click the red ball again to stop recording.
You just completed your first Web-Based automation! Congratulations!

What now?

Now you can repeat your automation steps as much as you want!

You may even have a temptation to move on with selenium. Great!

Your "test case" or rather "automation" can be exported into different programming languages i.e. C#, Java, Python, Ruby.

How to export your Selenium IDE recorded session?

Go to File -> Export Test Case As -> (and there you pick one of your favorite programming languages) i.e. Python2/unittest/Webdriver.

Last but not least

This was the first article about Selenium. Be prepared for more!

Thanks - and a bonus!

Today it was a very short article - more a simplified tutorial.

For make it more juicy I'm adding my script that I did today to check if it's even possible.

A PacktPub Free Learning Claiming Automation Script - just change login and password in script and you good to go.

PacktPub Free Learning Selenium IDE

ListSerializers - Django Rest Framework

2017-07-21T00:20:00+02:00

DRF - what is that?

It's a simplifier for creating API using Python's Django framework.

You can check my short comparison between different type of Django-supported API frameworks here

Django Rest Framework example project

Initially, I wanted to make my own sample Django REST Framework project that will be used in this tutorial.

Then I've thought to myself- "There must be a better way" (quoting Uncle Bob ^^) - so I've found a repository at GitHub with already predefined simple logic for maintaining TODO-list as an REST-api.

Check it here.

DRF Environment setup

Check out my prepared GitHub-Gist with a simple Linux-commands that you use to prepare your environment HERE.

Checking out the source code of sample project for bulk todo-elements creation.

So first things first - let's check if our code will enable use creation of more than one element using API at once. Let's make a POST request with a list of todos:

[
    {"name":"TEST1"},
    {"name": "TEs<t2"}
]

The output of it is :

{
    "non_field_errors": [
        "Invalid data. Expected a dictionary, but got list."
    ]
}

Means - no. No luck this time :) Let's un-release this and make a bulk of TODOs per one request.

List Serializer

I'm writing this post-heading information after about 3 hours. I've struggled to work with List-Serializer. I tried to use it in a "django" way - but for now, it's beyond my perception. I've found a solution at stackoverflow that after few adoption tricks started to work properly. It's not perfect - but quoting "Done is better than perfect" - I consider it a work-in-progress solution.

I might go back to this and try to investigate how it should be done in more "Django" way

Checkout code:

class TodoBatchAPIView(ListCreateAPIView):
    """
    API endpoint that allows multiple members to be created.
    """
    queryset = Todo.objects.none()
    serializer_class = TodoSerializer

    def get_queryset(self):
        queryset = Todo.objects.all()
        return queryset

    def create(self, request, *args, **kwargs):
        serializer = self.get_serializer(data=request.data, many=isinstance(request.data, list))
        serializer.is_valid(raise_exception=True)
        todo_created = []
        for list_elt in request.data:
            todo_obj = Todo.objects.create(user=request.user, **list_elt)
            todo_created.append(todo_obj.id)
        results = Todo.objects.filter(id__in=todo_created)
        output_serializer = TodoSerializer(results, many=True)
        data = output_serializer.data[:]
        return Response(data)

and in urls.py:

url(r'^batch/$', TodoBatchAPIView.as_view(), name="batch"),

Thanks :)

Hope you find this solution valuable.

I for sure will use it in my Biking-Endorphines project.

Please comment and feel free to post your solution to this problem!

Biking Endorphines - Points Serializer!

2017-07-16T15:10:00+02:00

Long Time No See

It's been a long time since I've been looking into this project from code-perspective.

I needed few deeper moments with it to find myself again in it after those 2 months (woooot??) - yeah...

What task should I get ?

That's the first thing I've asked myself.

I've grabed github issues to investigate on which task would not need so much time.

Initially I've started looking at issues estimated for 2h.

I have found issue on which I was working on - the #55 - subtask of #46 which in fact is also a subtask of #26.

It was not foreseen by me at first glance - initially I've started working on #55 - then I've found out about #46 from commits in repository - Then I've found what problem I've been struggling with last time.

The infamous File-Upload with Route-information.

This time I've decided to move with different approach - step-by-step - not all-in-one :)

To the #56 !

After re-thinking this problem I've found out that without Point serializer I will not be able to create Route-serializer.

So I've changed my mind and instead of working on #46 I went with making Point-Serializer instead of resolving issues with File-Uploading/Json Uploading( json uploading in fact is working properly - nothing to work on in that area)

Fixing time!

Code I've found did not work properly - it failed at 2 levels - both pylint's perspective and pytests were broken.

I decided to first fix the pytests perspective - cause fixing pylint's perspective would give a false-positive affect that It's done - which in fact could not be after making changes for pytests working properly.

Final result

Finally I've completed only part of Point-Serializer - for making only single point creation instead of bulk. I will find out what was wrong with my code next time.

Source Code:

1. In `api/serializers.py`

class PointListSerializer(serializers.ListSerializer):
    """
    Point List serializer
    """

    def create(self, validated_data):
        """
        Create and return list of a new `Point` instances, given the validated data.
        """
        points = [Point(**item) for item in validated_data]
        return Point.objects.bulk_create(points)

    def update(self, instance, validated_data):
        # Maps for id->instance and id->data item.
        point_mapping = {point.id: point for point in instance}
        data_mapping = {item['id']: item for item in validated_data}

        # Perform creations and updates.
        ret = []
        for point_id, data in data_mapping.items():
            point = point_mapping.get(point_id, None)
            if point is None:
                ret.append(self.child.create(data))
            else:
                ret.append(self.child.update(point, data))

        # Perform deletions.
        for point_id, point in point_mapping.items():
            if point_id not in data_mapping:
                point.delete()

        return ret



class PointSerializer(serializers.ModelSerializer):
    """
    Point serializer
    """
    class Meta:
        model = Point
        fields = ('id', 'lat', 'lon', 'elevation', 'time')
        list_serializer_class = PointListSerializer

    def create(self, validated_data):
        """
        Create and return a new `Point` instance, given the validated data.
        """
        return Point.objects.create(**validated_data)

2. In `web/models` added `time` field in Point.

Tests:

1. In `api/tests/test_integration.py`

class TestFileUploadView(APIGeneralTestCase):
    "File Upload tests"
    def setUp(self):
        super(self.__class__, self).setUp()
        self.api = APIRequestFactory().post
        self.view = FileUploadView.as_view()

    @unittest.skip("test not ready")
    def test_json_file(self):
        """
        This test has been setup as skipped temporarily - because it does not make what it should
        Tests if making api request for uploading json file succeed
        """

        import os
        dir_path = os.path.dirname(os.path.realpath(__file__))
        file_ = open(dir_path + '/samples/routesample.json', 'rb')

        api_data = {'filename':  file_}

        self.client.credentials(HTTP_AUTHORIZATION='Token {}'.format(self.token))
        request = self.api(
            "/api/file/",
            api_data,
            format='json',
            HTTP_AUTHORIZATION='Token {}'.format(self.token),
        )
        response = self.view(request, format='json')
        assert response.status_code == 202


class TestPointView(APIGeneralTestCase):
    """ Tests for Point View"""

    def setUp(self):
        super(self.__class__, self).setUp()
        self.api = APIRequestFactory().post
        self.view = PointView.as_view()
        self.path = "/api/point/"

    def test_uploading_one_point(self):
        " Tests if making api request updates user with new name"

        data = json.dumps(
            {
                "lat": "11.22",
                "lon": "11.33",
                "time": "2001-01-22T07:05:05Z",
                "ele": "11.1",
            }
        )
        response = self.get_response_user_api(
            data
        )
        self.assertEquals(response.status_code, 201)
        all_points = Point.objects.all()
        self.assertEquals(len(all_points), 1)


class TestPointListView(APIGeneralTestCase):
    """ Tests for List of Points View"""

    def setUp(self):
        super(self.__class__, self).setUp()
        self.api = APIRequestFactory().post
        self.view = PointList.as_view()
        self.path = "/api/points/"


    @unittest.skip("test not ready")
    def test_uploading_multiple_points(self):
        " Tests if making api request updates user with new name"
        data = json.dumps([
            {
                "lat": "11.22",
                "lon": "11.33",
                "time": "2001-01-22T07:05:05Z",
                "ele": "11.1",
            },
            {
                "lat": "22.22",
                "lon": "22.33",
                "time": "2001-01-22T07:05:05Z",
                "ele": "22.1",
            }
        ])
        response = self.get_response_user_api(
            data
        )
        self.assertEquals(response.status_code, 200)
        all_points = Point.objects.all()
        self.assertEquals(len(all_points), 2)

2. In `api/tests/test_unit.py`:

class TestPointSerializer(unittest.TestCase):
    "Point Serializer test"

    def test_point_serialize(self):
        "Tests if Point object serialize with PointSerializer"
        input_point = Point()
        input_point.lat = 33.2
        input_point.lon = 34.2
        input_point.elevation = 11.2
        input_point.time = "2017-03-03T12:00:00+00:00"
        input_serialized = PointSerializer(input_point)
        self.assertEqual(
            (input_serialized.data),
            {
                'id': None,
                'lat': 33.2,
                'lon': 34.2,
                'elevation': 11.2,
                'time': "2017-03-03T12:00:00+00:00"
            }
        )

    def test_point_list_serialize(self):
        "Tests if Point object serialize with PointSerializer"
        input_point = Point()
        input_point.lat = 11.1
        input_point.lon = 11.1
        input_point.elevation = 11.1
        input_point.time = "2011-01-01T11:00:00+00:00"

        input_point2 = Point()
        input_point2.lat = 22.2
        input_point2.lon = 22.2
        input_point2.elevation = 22.2
        input_point2.time = "2012-02-02T12:00:00+00:00"
        input_serialized = PointSerializer([input_point, input_point2], many=True)
        self.assertEqual(
            (input_serialized.data),
            [
                OrderedDict(
                    [
                        ('id', None),
                        ('lat', 11.1),
                        ('lon', 11.1),
                        ('elevation', 11.1),
                        ('time', '2011-01-01T11:00:00+00:00')
                    ]
                ),
                OrderedDict(
                    [
                        ('id', None),
                        ('lat', 22.2),
                        ('lon', 22.2),
                        ('elevation', 22.2),
                        ('time', '2012-02-02T12:00:00+00:00')
                    ]
                )
            ]
        )

Code commits done for this post:

Release:

16-07-2017-point-serializer

What else I've found out when looking into code and environment I'm currently working on?

1. Lack of debug mode when using tests which in fact may help to figure out problems with code.

2. Pylint's report contains too much information when developing and fixing - it leads to problems but it's too much of data (simple file line and error of that line would be enough)

3. Development environment for writing drafts of articles needs a bit of my attention to make it a bit simpler to use.

4. Figure out that I need a plugin in pelican that will automatically convert issues numbers into http-links to project in some fancy-way that I would only need to put the `[#46]` and it would be enough for pelican plugin to read it and convert to `[#46](https://github.com/anselmos/Biking-Endorphines/issues/46)`

5. Lack of a simplified check-list that I'll use each time writing blog post and developing b-endorphines.

To Be Continued - See you next time!

Time of coming back to blogging experience!

2017-07-09T13:00:00+02:00

Employer change!

I'm back into mode of blogging experience :) . It was a bit too long vacations from blogging, so I've almost forgot how to blog :)

I had to make this vacations from blogging to speedup learning new things to company that I've started working in this month.

Yeah- you guest it - I've changed my employer! It was a long journey with Mobica - and finally - after 4 years - I've decided to make changes.

I'm really greatfull for all the experience I've gathered there! Great people and great projects that I had a chance to work on.

There was also some less fun projects - but that's normal in our IT-industry. Unfortunatelly in my experience they are the onces that made me think about change.

Finally thanks to my a bit too high ambitions I've decided to move on and find new challenge. As far as this week went by in new work - I personally think it was good choice.

For now I find it both challengeful and joyful, so as a bit scary - but that's normal. We fear things that we kind-of should do. That's why we are scared of them - they present a challenge for us. Problems we will be fighting against will shape us.

I don't think all of us should go and challenge them-self. I just wanted to say - it works for me.

I personally think now that I'm not fearfull being. But being brave means to be scared but move on!

Just a bit about new work - I'm working as a python developer within project that uses framework much similar to django.

Changes part 2 :)

That's not all. I've decided to go the unknown path of using "3 levels of tasks"3 poziomy pl - thanks to Mirek! and check if it will lead me to any progress. And it kind of did.

Unfortunatelly while I tried to fulfill my task of "5 am domination of day" I got health problems and finally I got flu - and got myself banned from outside world for a week.

You may wonder how in the middle of summer I got flu? That's because I was not thinking straight :) I thought that I will just try all things that I got in my mind and bring them ASAP.

That's why my organism gave up - I did not sleep enough (means at least for 7-8 hours - i did about 5-6 hours) and also first thing I did after waking up was riding a bike and then having a shower. All of this would not be something bad - unfortunatelly one time temperature outside was not enough to wear a short outfit (shortened on shorts and shirt) - and that my shower was not a hot-shower, but a half hot half cold shower - and that got my organism finally to gave up. :( (LL - lesson learned. )

There was also a good side of 3-levels - after making each level of task you should give yourself something in reward (as Mirek says and does in his 3-levels - check his big list! ).

So I also did - After making progress in some fields (i.e. having Android badge ) - I reward myself - but in a way that will bring my attention to move with next challenges in 3-levels.

That's why I've bought 16GB of RAM to my Lenovo X220 that I currently use - check pictures:

Blogging against full-time job.

Taking into account that my current job requires a full attention from me - when I come back to home I kind-of don't have any more left energy to make any progress with anything that is combined with programming.

That means that probably I will make progress with my private projects in weeekends.

I hope to find solution to work on them also in work-week days - using some tricks like:

After come back from work go ride a bike for 30 minutes to get more relaxed.
Take a shower after going back to home to refresh myself ( self.refresh() :D )
Bring pomodoro technique and my todo-lists in life and try to make a least one additional thing per day. (to motivate myself)

Plans?

The only plan I have is to blog at least ones a week. It may be about my current project (BikingEndorphines) or my back-side projects or other things :)

Also I do hope, till end of this year I will manage to complete in some usable stage my current project (B-Endorphines) - finger crossed ! :)

THANKS!

Thanks for all of you that reads this blog - it encourages me to move on and make progress with priv. projects. :)

This time it was not a technical-stuff - but beaware - you may or maynot find next article more technical :) That's a surprise !

DSP Gala!

2017-06-18T14:30:00+02:00

Introduction

DSP Gala?

For everyone that does not know yet - DSP is a contest made by Maciej Aniserowicz in which one had a chance to Get Noticed, create a blog with at least 2 articles by week and creating open-sourced project at github.

It's the second edition of "Get Noticed" Gala.

The Agenda for the Gala event can be found in here - in polish lang.

My attendance in DSP Gala?

I have been one of the finalist of DSP2017 (the one that have made 2 articles per week).

I've decided to meet Maciej Aniserowicz and other participants. Also I had high hopes to introduce myself and my project to others in some way.

From Paris to Berlin... I mean... From Bydgoszcz to Warsaw

So to attend in gala, I had to either drive by car, take a bus or use train.

I've decided to go by train - because after checking arrival time - I had high hopes not to be late :)

Sure thing - I was not - I managed to be at registration short before ending :)

Handshaking Maciej Aniserowicz!

So that was first time I had a chance to meet Maciej!.

It was a great opportunity for me to work on my introvertism and hand-shake his hand!

And I did! Man - that was so liberating, I feel so good even now!

DSP Participants!

Wow ! That was my first shock when I've arrived at Warsaw with my colleague - Wojciech - At the Gala there was a full-house of people. I knew that there will be around 150 people, but I did not anticipate almost all of them will come!

Since Maciej was so generous and add information that no-formal-style was needed - All of us could feel more comfortable in our usual daily outfit :)

I had high hopes to get to know other people and participants that I compete against :)

From first glance I knew there will be a lot of Dot-Net programmers - but I found a lot of programmers that knew more then Dot-Net. I even found a pythonist! - High-Five to you Jakub

I even found one of them got more interested in my project ! Fantastic! My goal has been achieved!

DSP participants and their presentations!

Participants of DSP had a chance to have their presentation at Gala.

Applause for all of them! I personally had a lot of fun on each and every one!

Even my colleague had his presentation! The “Daj się poznać’ się skończyło. Twoja przygoda z blogowaniem dopiero się rozpoczyna” - I had to admit Wojciech - you did great job! You have motivated me personally to move on with project!

Afterparty

After DSP there was an afterparty. Unfortunatelly I could not participate in it. I hope to find all of you at other conferences and meetups!

Blog and What now?

Lately I tried to make a plan of creating 2 articles in week.

That did not worked out because of my learning to work and also because of my weekend Android-Google-Exam that I passed!

See my my badge:

I've decided that for next 2 weeks (till end of June) I will not blog. I had to focus on learning stuff needed in work.

After that time, I hope to blog at least once a week.

So let's see you in July! Thanks!

Biking Statistics - Who Cares about Them actually?

2017-06-04T20:30:00+02:00

What are statistics ?

Statistics are just numbers that are collected (in biking context) from variety of devices, starting from simple count-down timer and ending on more technology advanced collectors like Endomondo Android application, bicycle counter with gps/heart rate/elevation and prognosis of progress.

Based on data that you can obtain, you may create different forms of statistics: - Graphs - Diagrams - Counters i.e. average m/h(miles per hour)

Also notable would be to acknowledge statistics data: - Simplest average km/miles per race-route - Highest/Lowest elevation of route - Heart Rate levels on route ( i.e. in which points of route you get tired - which gives feedback on what elements of route (mounting) you could level-up) - many more that I did not yet discovered

Why are they important ?

Pros:

Motivation - you obtain knowledge that you otherwise would not gain with which you are getting more motivation to bring more from yourself!
Measure progress - especially needed within racing/trainings.
Focusing on your goals! - you can focus on specific goal - i.e. to make next lap in training faster then before.

Cons:

You may get dismotivated ! - if you will not have progress with your goals,
It's not for everyone - short term bikers should first learn to enjoy making best of biking instead of focusing on numbers.
Statistics can be bound to your matter - you may start to fool yourself by using statistics (as they say - "There's truth, semitruth and statistics" )

Who really cares about statistics ?

In short answer to this is simple - anyone who wants to achieve something in sport - it does not matter if its cycling, athletics, swimming or running.

Elaborating, in each of this sports you use statistics and numbers to make competition between each participants.

This then makes a good ground to motivate players to level-up and be better than your competition and yourself sometimes (beat your own records).

I myself in particular find statistics fascinating and motivational. They not only gives me a feedback about how my body feels and where it stands, but also describes my successes and failures.

But there is also the second side of the coin - sometimes I find statistics in applications - especially combined with competition - a bit demotivational. Taking an example - if you don't have so much time in some month as you had in previous, then you see lower progress against opponent then you expected it to be. And it should be perfectly fine, because you didn't have time, but still statistics don't have that accounted.

Why our Biking Endorphine project uses statistics??

First of all when I've created this project initially it was called "biking-stats" - that means I wanted to achieve a better statistics than there is in free-mode at Endomondo. Let's look at arguments against and for statistics in project:

For:

You have real feedback from gps about progress of each user
You can count information about such user and give them feedback - since each and almost every sport-applications does.

Against:

As previously said statistics can lie to you in some way
Using statistics with competition creates a winner/looser situation, where none of them are good way to achieve more(in a positive way)

Thanks

Thanks folks, that's all. In the comments below, you can give me a feedback what do you think about statistics in our sport-life and programming?

Do you find them atractive and positive ? Do you care about them ??

Maybe not and you struggle against them ? Any comment is appreciated.

See you soon :)

Blogging and Final DSP article - Bye Bye to DSP

2017-05-31T15:50:00+02:00

Why did you start Blogging?

Before I've decided to get involved in blogging, I've thought - I don't have time for this. I should be learning, programming, but not blogging.

Where is fun in that?

For more then few years I did not blog ( I had an internal blog - one that does not qualify as a blog, but more for a note-taking place about technical stuff). It's not that I haven't been thinking about it, but I didn't considered it as a way for making more progress in programming.

Years passed by and finally I've found myself in work at project in which I did not develop any features. I thought to myself - just get over with it, it will pass, someday you will program new things. But day after day, week after week I've struggle with not programming and my frustration levels went far beyond any understanding.

Then I've though - it's time to get back on my biking-trainings at mornings that always gave me such a pleasure for almost all day.

It worked partially. At the final stage I've decided to go with something more than just relaxing myself before work!

Strike 1 - online-courses!

Online courses worked out for few weeks. But not for long. I didn't want to spend my time only on some useless courses that didn't bring any potential measure outcome in meaning - a project that I could maintain.

Strike 2 - power of conferences!

That was my second and more successful attempt to get out of comfort zone. That worked very well - not only I had to overcome my shyness to new people, but also I could learn something new.

Yeah you could just play youtube-channel with some conference, but that is just a substitute of how much you can gain going on such event!

The BoilingFrogs conference on which my friend Wojciech asked me to go, was the one that inspired me a lot.

It was that much of inspiration, that I've decided to start my own Blog. I will not say about "Why you should have your own blog" - there has been a lot of articles out there about that and youtube-videos that I would just repeat other's thoughts.

To simply summarize why I started blog - for learning new things, making articles that could be useful for potential future audience. Learn by explaining!

DSP and final thoughts

At the BoilingConference I've found out about DSP contest. It was 25 of February. Just few days before end of registration for this contest. I didn't know back then If I want to participate in such a big thing. Initially I've started my blog without DSP - with my simple Wednesdays Tutorials article.

Few days later at 2nd of March I read information that registration has been extended for another 10 days - And thought to myself - what's the worse that can happen? Even if I fail to make project in that time - I will learn new things that may push myself away from comfort zone!

And I've created my first article with DSP tag

For project I've figure out - why not restart my long waiting project of Biking statistics that I've kind-of abandoned.

Within this project I've made a lot little successes that I did not anticipate before making it! So yea... it worked out! A lot! Thanks Maciej that I could participate in this contest!

Accomplished at Biking-Endorphines Projects!

Originally I thought I will create both Android application and python backend with web module.

That was not realistic thinking taking into consideration my previous duties (full-time job, weekend course and other). Fortunatelly I've cut-off Android-part somewhere in the middle of April.

Now to the optimistic part :)

I've accomplished:

GitHub project created
Integration with Travis build and having pylint&tests integrated
Integration of pyreverse to create live diagrams with Travis build
Source code for Rest-API
Initial source code for endorphine-algorithms.
TDD in mind :)
Github Issues Closed
Github Pull Requests Closed

Overall - I've done pretty much good job. I know I could have done more - but as I've recently said in one of my posts - Done is better then perfect!

To Be continued

No, it's not over :) Projects will be continued.

Plans - What's next?

I plan to have at least 2 articles per week.

First article is going to be strictly Tutorials - where I will discover new things that may/may not be of any usefulness :)

Second article will be about Biking Endorphines project!

Thanks to all my readers! And ofcourse , Monty Python !

File Uploading at REST API with JSON Parsing into list of route-Points

2017-05-30T23:50:00+02:00

Custom JSONParser - extending Django-Rest-Framework JSONParser?

Why would you even need to extend JSONParser ? Well... I want to have a list of Points at the end of parsing, not a json-type data.

In that way I can easily manipulate this data and check information on it, i.e. using endorphine-algorithms to find out if badge can be applied to specific route!

How to start working on custom Parser from Djagno-Rest-Framework?

TDD approach of-course! But how? Well I didn't know myself when I started to figure it out, but then I thought to myself- why not check at source-code of Djagno-Rest-Framework - Django-Rest-Framework Parsers Test-case- But unfortunatelly this code does not provided me any valid answers.

Then I've looked into source code of parser and realized that the parse method is the one used. So I've went that way to extend functionality with mine.

I've firstly made an test for checking if only extending JSONParse without changing content and picking parse method will create at the end the same output as input (as a StringIO stream).

Then I've changed tests so they fit my assumptions about behaviour of this custom JSON parser.

Initial tests with source-code:

class RouteJSONParser(JSONParser):
    """ Route JSON Parser - parses JSON file into list of Points"""
    pass


class TestRouteJSONParser(TestCase):
    " Tests for RouteJSONParser class parse method overloaded"

    def setUp(self):
        self.parser = RouteJSONParser()

    def test_parse(self):
        " tests parser "
        input_dict = [
            {
                "lat": "53.126699",
                "lon": "18.072322",
                "time": "2000-01-01T07:47:02Z",
                "ele": 107.0
            },
            {
                "lat": "53.126699",
                "lon": "18.072322",
                "time": "2001-01-01T07:47:02Z",
                "ele": 99.0
            }
        ]
        input_stream = StringIO(json.dumps(input_dict))
        output = self.parser.parse(input_stream)
        assert str(output) == str(input_dict)

Final Source code of RoutesJSONParser with Tests

Tests:

class TestRouteJSONParser(TestCase):
    " Tests for RouteJSONParser class parse method overloaded"

    def setUp(self):
        self.parser = RouteJSONParser()

    def test_parse(self):
        " tests parser "
        input_dict = [
            {
                "lat": "53.126699",
                "lon": "18.072322",
                "time": "2000-01-01T07:47:02Z",
                "ele": 107.0
            },
            {
                "lat": "53.126699",
                "lon": "18.072322",
                "time": "2001-01-01T07:47:02Z",
                "ele": 99.0
            }
        ]
        expected = [
            Point(
                lat="53.126699",
                lon="18.072322",
                time="2000-01-01T07:47:02Z",
                ele=107.0
            ),
            Point(
                lat="53.126699",
                lon="18.072322",
                time="2001-01-01T07:47:02Z",
                ele=99.0
            )
        ]

        input_stream = StringIO(json.dumps(input_dict))
        output = self.parser.parse(input_stream)
        assert str(output) == str(expected)

Source-code - that resides in a test-module for now. I will definitely move that into source-files.

# pylint: disable=too-few-public-methods
class Point(object):
    """ Point object with lat/lon time and ele """

    def __init__(self, lat, lon, time, ele):
        self.lat = lat
        self.lon = lon
        self.time = time
        self.ele = ele

    def __repr__(self):
        return "{},{},{},{}".format(
            self.lat,
            self.lon,
            self.time,
            self.ele,
        )


# pylint: disable=too-few-public-methods
class RouteJSONParser(JSONParser):
    """ Route JSON Parser - parses JSON file into list of Points"""

    def parse(self, stream, media_type=None, parser_context=None):
        json_content = super(RouteJSONParser, self).parse(stream, media_type, parser_context)
        points = []
        for point in json_content:
            points.append(Point(point['lat'], point['lon'], point['time'], point['ele']))
        return points

File uploading via REST-API.

I wanted to follow TDD approach at this, but first I had to check how to create FileUploading solution cause I don't remember when I've done such thing!

So I've broke my "first test than production code", and first created a production code to check how it will look like, So I could find out how I should test it.

Fortunatelly for me Django-Rest-Framework Documentation on that subject.

After struggling for few hours with having still a "Not found" feedback information from source-code in tests, I've passed for now.

I'll figure this out someway, but this will not be today unfortunatelly :(

When checking manually source-code seems to be working, but I'm not sure if it does in all cases because of lack of passing tests... :/

Source-Code:

class FileUploadView(APIView):
    " A view for file upload "
    parser_classes = (JSONParser,)

    def post(self, request, format='json'):
        return Response(request.data)

Tests that is not passing yet:

class TestFileUploadView(APIGeneralTestCase):
    "File Upload tests"
    def setUp(self):
        super(self.__class__, self).setUp()
        self.api = APIRequestFactory().post
        self.view = FileUploadView.as_view()

    def _create_test_file(self, path):
        f = open(path, 'w')
        f.write('test123\n')
        f.close()
        f = open(path, 'rb')
        return {'datafile': f}

    def test_update_user(self):
        " Tests if making api request updates user with new name"

        import os
        dir_path = os.path.dirname(os.path.realpath(__file__))
        file_ = open(dir_path + '/samples/routesample.json', 'rb')

        api_data = {'filename':  file_ }

        self.client.credentials(HTTP_AUTHORIZATION='Token {}'.format(self.token))
        request = APIRequestFactory().post(
            "/api/file/",
            api_data,
            format='json'
        )
        response = self.view(request, format='multipart')
        assert response.status_code == 200

Bugs /Features I've found while working

Bugs:

Non so-far.

Features:

Parser for Routes in JSON data format #55

Code commits done for this post:

In branch feature/55-parser-for-routes-in-json-data-format: - Adds tests with RouteJSONParser

In branch feature/46-Endpoint-for-adding-new-routes-should-take-json-content-in-body:

Acknowledgements

StackOverFlow:

Other:

Use Postman for HTTP Requests

Djagno-Rest-Framework Documentation:

JSONParser

What's next

Contest is almost over. But working on project is not :)

Anndd.....

Monty Python is here again!

Thanks! See you soon!

Splitting bigger issues into smaller & Endorphine Badges Acquired

2017-05-28T18:40:00+02:00

Splitting big tasks into smaller one!

So I've been thinking, why I'm so not productive lately - and I've found reason - I've been scared of big tasks that consumes a lot of time!

I've decided to spend at least one Pomodoro timer (25 minutes) for splitting issues on Biking-Endorphines project.

At least ones that are estimated on more then 4 h.

Endorphine-Badges-Acquired #23

While reading this issue I've found it's not properly specified and should be splitted into smaller issues/features to make it more "pomodoro-a-like".

So I've added additional 2 tasks that helped me a lot to make progress with this issue.

Subtasks of #23 : #44 and #45

Endpoint for gathering all acquired badges needed 2 models with following them tables : - Badges - UserBadges

And then I could work on creating endpoint with serializer.

So I've created corresponding [REST-API] Badges Model needed #44 and [REST-API] Badge-To-User model #45 and described them as specific as I could.

Pomodoro Technique Timer from Toggl :)

I've also recently started testing new platform called Toggl - for tracking time.

It has such a fancy-pancy chrome-addon that helps to track of time for specific github-issue and has build-in pomodoro timer that after 25 minutes can trigger notification from browser.

Using it helped me with "high-density" working on issues above.

Bugs /Features I've found while working

I've found few bugs/features that I'll be working on soon after this Python-Milestone will end.

Bugs:

Features:

The #47 and #46 were put into Python-Milestone because they are closely related with #26

Final version

Test code:

at api/tests/test_unit.py:

class TestUserBadgeSerializer(unittest.TestCase):
    "UserBadge Serializer test"

    def test_user_badge_serialize(self):
        "Tests if user_badge object serialize with RouteSerializer"
        user = User()
        user.id = 21
        user.name = "Anselmos"
        user.surname = "Somlesna"
        user.weight = 80
        user.height = 175

        badge = Badge()
        badge.id = 1
        badge.name = 'FasterThenUniverse'
        badge.description = 'you\'re average lap was faster then previous one'

        route = Route()
        route.id = 2
        route.route_name = "NameOfThisFancyRoute"
        route.avg_route = 19.9


        input_user_badge = UserBadge()
        input_user_badge.id_user = user
        input_user_badge.id_badge = badge
        input_user_badge.id_route = route
        input_user_badge.active = True
        input_user_badge.badge_acquiring_date = "2017-03-03T12:00:00+00:00"
        input_user_badge.activation_modification_date = "2017-03-03T12:00:00+00:00"

        input_serialized = UserBadgeSerializer(input_user_badge)
        self.assertEqual(
            (input_serialized.data),
            {
                'id': None,
                'id_badge': 1,
                'id_user': 21,
                'id_route': 2,
                'active': True,
                'badge_acquiring_date': '2017-03-03T12:00:00+00:00',
                'activation_modification_date': '2017-03-03T12:00:00+00:00',
            }
        )

at api/tests/test_integration.py::APIGeneralTestCase

    # pylint: disable=no-self-use
    def create_badge_in_db(self):
        " Creates new badge object in database "
        return Badge.objects.create(
            name='FasterThenUniverse',
            description='you\'re average lap was faster then previous one'
        )

    # pylint: disable=no-self-use
    def create_route_in_db(self):
        " Creates new route object in database "
        return Route.objects.create(route_name="NameOfThisFancyRoute", avg_route=19.9)

    # pylint: disable=no-self-use
    # pylint: disable=too-many-arguments
    def create_user_badge_in_db(self, user, badge, route, active, badge_date, activation_date):
        " Creates new UserBadge object in database "
        return UserBadge.objects.create(
            id_user=user,
            id_badge=badge,
            id_route=route,
            active=active,
            badge_acquiring_date=badge_date,
            activation_modification_date=activation_date,
        )

at api/tests/test_integration.py

class TestUserBadgeList(APIGeneralTestCase):
    "User Update tests"
    def setUp(self):
        super(self.__class__, self).setUp()
        self.view = UserBadgeList.as_view()

    def test_update_user(self):
        " Tests if making api request updates user with new name"

        badge = self.create_badge_in_db()
        route = self.create_route_in_db()
        user = self.create_user_in_db()
        date_badge_acquired = "2017-02-02T22:00:00"
        date_badge_activation = "2017-02-02T22:00:00"
        user_badge = self.create_user_badge_in_db(
            user,
            badge,
            route,
            True,
            date_badge_acquired + "+00:00",
            date_badge_activation + "+00:00"
        )

        self.path = "/api/badge/{}/".format(user_badge.id)
        response = self.get_response_user_api(
            pk_id=user.id
        )
        self.assertEquals(response.status_code, 200)
        self.assert_user_badge(
            response.data[0],
            user.id,
            badge.id,
            route.id,
            user_badge.active,
            unicode(date_badge_acquired + "Z"),
            unicode(date_badge_activation + "Z")
        )

    # pylint: disable=too-many-arguments
    def assert_user_badge(
            self, user_badge_object, user_id, badge_id, route_id,
            active, badge_acquiring_date, activation_modification_date):
        """ Asserts user badge object with parameters """

        self.assertEquals(user_badge_object['id_user'], user_id)
        self.assertEquals(user_badge_object['id_badge'], badge_id)
        self.assertEquals(user_badge_object['id_route'], route_id)
        self.assertEquals(user_badge_object['active'], active)
        self.assertEquals(
            user_badge_object['badge_acquiring_date'],
            badge_acquiring_date
        )
        self.assertEquals(
            user_badge_object['activation_modification_date'],
            activation_modification_date
        )

Source Code:

at web/models.py:

class Badge(models.Model):
    "Badge Model"
    name = models.CharField(
        max_length=50,
        help_text="describes in short name of the badge"
    )

    description = models.CharField(
        max_length=256,
        default="",
        help_text="More thorough information about badge acquiring (i.e. criteria)"
    )

    def __unicode__(self):
        "Returns Badge unicode as name: description "
        return "{}: {}".format(self.name, self.description)


class UserBadge(models.Model):
    "Users acquired Badges!"
    id_user = models.ForeignKey(User, on_delete=models.CASCADE)
    id_badge = models.ForeignKey(Badge, on_delete=models.CASCADE)

    id_route = models.ForeignKey(
        Route,
        on_delete=models.CASCADE,
        help_text="id of route on which badge has been acquired"
    )

    active = models.BooleanField(default=True)
    badge_acquiring_date = models.DateTimeField()
    activation_modification_date = models.DateTimeField()

    def __unicode__(self):
        "Returns Badge unicode as name: description "
        return "User : {},  Badge acquired-id: {}: "\
        "Route-id: {}, active?: {}, badge_acquired_date: {}, "\
        "activation_modification_date: {}".format(
            self.id_user,
            self.id_badge,
            self.id_route,
            self.active,
            self.badge_acquiring_date,
            self.activation_modification_date
        )

at api/serializers.py:

class UserBadgeSerializer(serializers.ModelSerializer):
    """
    User serializer
    """
    class Meta:
        model = UserBadge
        fields = (
            'id',
            'id_badge',
            'id_user',
            'id_route',
            'active',
            'badge_acquiring_date',
            'activation_modification_date'
        )

    def create(self, validated_data):
        """
        Create and return a new `UserBadge` instance, given the validated data.
        """
        return UserBadge.objects.create(**validated_data)

at api/views.py:

class UserBadgeList(generics.ListCreateAPIView):
    """
    get:
    Return a list of all existing user badges.

    """
    queryset = UserBadge.objects.all()
    serializer_class = UserBadgeSerializer

Code commits done for this post:

In branch feature/44-Badge-model-needed:

In branch feature/45-Badge-to-User-model:

In branch feature/23-Endorphine-badges-acquired:

Release:

28-05-07-badges-acquiring-api

What's next

Hunting never ends!! .

I'm still on my python-hunting for collecting as much of Python Milestone Issues as I can :

Due by May 31, 2017 66% complete (9 Open/ 18 Closed)

Thanks! See you soon!

Biking Endorphines - Zero Division Error

2017-05-24T18:30:00+02:00

Zero Division Error bug #40.

So yesterday I've started working on this bug. I've estimated it for about 1h.

It was fortunatelly a very realistic estimation. When making changes and creating Test-case for this issue, I've found myself thinking, is it really possible to use BMI health check for people who are less then 1m hight.

So I've looked at internet and Voilà! Other calculators do that! So without struggle I've adapted my code to be able to put less height then 1m long.

I know it may sound like a trifle, but I've thought that people using this application will probably not lie to themselves and make a fake data.

What was wrong with my code?

My assuption is that height is setup in centimeters.

That's why I've fallen into situation where I need to change floating point number to adjust formula not to return zeros.

Test source code:

    def test_height_less_hundred(self):
        """ tests for height less then hundred if not raises ZeroDivisionError"""
        self.user.height = 99
        self.user.weight = 50
        self.user.name = "under hundred height"
        self.user.surname = "under fifty weight"

        self.assertEquals(self.user.bmi(), 51.02)

BMI method source code:

As you can see by code below, I've decided to change all the parameters inside of bmi method into float's .

It's not best, because I could change them at the initialization part, but still - *Done is better then perfect!*

    def bmi(self):
        """
        Body Mass Index calculator simplified to number
        """
        return round((float(self.weight) / (float(self.height)*float(self.height))) * 10000, 2)

Code commits done for this post:

Acknowledgements

What's next

I'm going to continue my hunt for Python Milestone Issues

Thanks! See you soon!

Biking Endorphines - Python Web Server with API

2017-05-21T19:00:00+02:00

Status of project

As promissed I'm giving information on how is the progress with features/bugs:

Milestone update.

As you may noticed it's also the same info as in the Python Milestone that I've recently updated due-day to 31 of may.

I do hope this timing will be more realistic and more realistic estimated then what it was at previous stages!

Tools and applications used:

Thanks

Biking Endorphines - Plans that were not Real-Life correlated.

2017-05-20T18:00:00+02:00

What happen?

Let's start from last post about the plans - I did terrible thing of a bit giving up. And the only one to blame is myself.

I've planned to produce till end of this week a python-part.

My plan was too much optimistic that I'll have at least 1-2 hours per day to make commits and release code. Also this features that I had to create were not well documented of what I'm suppose to actually do.

Unfortunatelly there was few things that I did not anticipate in my plan - i.e. going to JUG conference.

Lately I've found a youtube clip from Tim Ferris about productivity and procrastination that made me more optimistic about future!

He says that every task should be splitted intochunks. Then you need to make estimation of how much of this tasks you can do per day - and while making that you should make sure that it's less that you know you could produce - in that way you will everyday pass your task by making this chunk of task and if lucky you may notice yourself exceeding this limit.

What now ?

From now I will not make any plans that are not at least estimated and prioritized.

Let's find out if Github Issues can be handled with estimation points ?

After 10 minutes of searching - Well there's no obvious option like "it will take me 1 hour of work".

Unfortunatelly Github is not like a Redmine or Trac that gives option to setup things like that.

Let's for now use labels instead.

Raise Up like Phoenix!

Yes! I'm now going to raise up and make an effort on focusing on python backend/rest-api part till end of May.

Then go back to Android part, create simplest application that will bind with python-api and give end-user information about biking-routes/endorphines.

Final word

Now... I'm going to change issues and add labels. :)

Less talking - more coding! :D

BTW: I've lately started working on pygithub_playground for gathering github-issues - Checkout how fast and easy it is to get github issues from python command line!

Thanks!

See you soon!

Biking Endorphines - User API Serializer

2017-05-13T20:30:00+02:00

User Model and it's failure

I've started working on Users-API feature #22. Found that the Token-Authentication blocked api-docs - so I've created bug #39. I've temporarily commented out Token-Authentication and begun my work on User-API.

Then to my surprise ... User's API didn't properly handle bmi and the bmi-health-name.

So I've figure out it had to be some issue on Serializer site or on the QuerySet.

But no such luck, looking up this in internet lead me to some strange source-code changes that did nothing.

So I've dig a little deeper and debug the BMI algorithm itself. And voila! There was the problem!

As it turns out, yesterday I've made some testing for User model and for bmi-health-name function. While making that, I've used Mocking - which is in a general way a good idea. But to make use of Mocking you should be sure (having proper working tests!) that method/function you are mocking will output proper value. I didn't learn that lesson properly :)

So the BMI-Algorithm returned 0 all the time. When I've figure out where was the problem all other elements start ticking as they should !

As it turns out - I didn't learned lesson about making Unit-Tests independent from source-code - why ? Because of my laziness I've put some math-algorithm inside of test - instead of using hard-coded value.

When I've finally found it out- I've change code and then tests started to fail with wrong output as expected. I've fixed that :)

Now what about that User API ?

Well, when changing all of this things I've said before, I've changed the standard Serializer into a ModelSerializer. I also added Meta as described in ModelSerializer.

Then I've found that there is not tests for User Create/Update/Delete and retrieving list is not handled as it should.

So I've put effort into making best of this tests that are a coverage for User-API.

I also found bug #40 - when making tests for updating user.

Final result

Source Code of User API model:

class User(models.Model):
    """
    User model for obtaining personal information about biking riders
    """
    name = models.CharField(max_length=50)
    surname = models.CharField(max_length=100, default="")
    weight = models.IntegerField(default=0)
    height = models.IntegerField(default=0)

    def __unicode__(self):
        """
        Returns User information when using str/printing
        """
        return self.name

    def bmi(self):
        """
        Body Mass Index calculator simplified to number
        """
        return self.weight / ((self.height * self.height) / 10000)

    def bmi_health_name(self):
        """
        BMI Health Name - Returns proper naming for value of BMI
        """
        if self.bmi() < 0:
            return None
        if self.bmi() >= 0 and self.bmi() <= 18.5:
            return "Underweight"
        if self.bmi() > 18.5 and self.bmi() <= 24.9:
            return "Normal weight"
        if self.bmi() > 24.9 and self.bmi() <= 29.9:
            return "Overweight"
        if self.bmi() > 29.9:
            return "Obesity"

Source Code tests of User API:

"""
Tests for API
"""
import unittest
import json
from api.serializers import UserSerializer
from api.views import UserList, UserDetail
from web.models import User
from django.contrib.auth.models import User as AuthUser
from rest_framework.authtoken.models import Token
from rest_framework.test import APIClient
from rest_framework.test import APIRequestFactory
from rest_framework.test import force_authenticate


class APIGeneralTestCase(unittest.TestCase):
    "General Test Case for API"

    def setUp(self):
        self.path = "/api/user/"
        self.client = APIClient()
        self.user = AuthUser.objects.create_superuser('admin', 'admin@admin.com', 'admin123')
        self.token = Token.objects.get(user_id=self.user.id).key
        self.api = APIRequestFactory().get
        self.view = UserList.as_view()

    def tearDown(self):
        "Removes all AuthUser and Users objects at the end of each test"
        self.user.delete()
        User.objects.all().delete()

    def get_response_user_api(self, api_data=None, pk_id=None):
        " Creates response for /api/user List get with forcing login with Token-Authentication "
        self.client.force_login(user=self.user)
        request = self.api(
            self.path,
            api_data,
            HTTP_AUTHORIZATION='Token {}'.format(self.token),
            content_type='application/json'
        )
        force_authenticate(request)
        if pk_id is not None:
            return self.view(request, pk=pk_id)
        return self.view(request)

    def assert_status_code(self, response):
        " Asserts response status code with 200"
        self.assertEquals(response.status_code, 200)

    # pylint: disable=no-self-use
    def create_user_in_db(self):
        " Creates new user object in database "
        return User.objects.create(name='Bart', surname="Trab", weight=80, height=175)

class TestUserSerializer(unittest.TestCase):
    "UserSerializer tests"

    def test_user_serialize_to_json(self):
        "test if serializing to JSON works"

        mocked = User()
        mocked.name = "Anselmos"
        mocked.surname = "Somlesna"
        mocked.weight = 80
        mocked.height = 175

        user_serialized = UserSerializer(mocked)
        self.assertEqual(
            (user_serialized.data),
            {
                'height': 175,
                'surname': u'Somlesna',
                'id': None,
                'weight': 80,
                'name': u'Anselmos',
                'bmi': 26,
                'bmi_health_name': u'Overweight'
            }
        )


class TestUserList(APIGeneralTestCase):
    "UserList tests"

    def test_user_get_return_json(self):
        "test if using get returns json data"

        self.client.force_login(user=self.user)
        response = self.get_response_user_api()
        self.assert_status_code(response)

    def test_user_list_return_json_list(self):
        " Test if setting up user returns user list"

        input_user = self.create_user_in_db()

        self.client.force_login(user=self.user)
        response = self.get_response_user_api()
        self.assert_status_code(response)
        self.assertEquals(len(response.data), 1)
        self.assert_user_object(response.data[0], input_user)

    def test_get_two_user_list(self):
        " Test if setting up user returns user list"

        input_user = self.create_user_in_db()
        input_user2 = User.objects.create(name='B222art', surname="Trabaaaa", weight=50, height=100)

        response = self.get_response_user_api()
        self.assert_status_code(response)
        self.assertEquals(len(response.data), 2)
        self.assert_user_object(response.data[0], input_user)
        self.assert_user_object(response.data[1], input_user2)


    def assert_user_object(self, input_json, expected):
        "Asserts response User input json object == expected user model object. "
        self.assertEquals(input_json['name'], expected.name)
        self.assertEquals(input_json['surname'], expected.surname)
        self.assertEquals(input_json['weight'], expected.weight)
        self.assertEquals(input_json['height'], expected.height)
        self.assertEquals(input_json['bmi'], expected.bmi())
        self.assertEquals(input_json['bmi_health_name'], expected.bmi_health_name())


class TestPostUser(APIGeneralTestCase):
    "User Post tests"
    def setUp(self):
        super(self.__class__, self).setUp()
        self.api = APIRequestFactory().post

    def test_return_user_object(self):
        " Tests if making api request returns user object "

        data = json.dumps(
            {"name":"Test", "surname":"tester1", "weight":88, "height":173}
        )
        response = self.get_response_user_api(data)
        self.assertEquals(response.status_code, 201)


class TestDeleteUser(APIGeneralTestCase):
    "User Delete tests"
    def setUp(self):
        super(self.__class__, self).setUp()
        self.api = APIRequestFactory().delete
        self.view = UserDetail.as_view()

    def test_delete_user(self):
        " Tests if making api request deletes user with 204"

        user = self.create_user_in_db()
        self.path = "/api/user/{}/".format(user.id)
        response = self.get_response_user_api(pk_id=user.id)
        self.assertEquals(response.status_code, 204)


class TestUpdateUser(APIGeneralTestCase):
    "User Update tests"
    def setUp(self):
        super(self.__class__, self).setUp()
        self.api = APIRequestFactory().post
        self.view = UserList.as_view()

    def test_update_user(self):
        " Tests if making api request updates user with new name"

        user = self.create_user_in_db()
        self.path = "/api/user/{}/".format(user.id)
        response = self.get_response_user_api(
            json.dumps({"name": "newName", "weight": 88, "height": 111}),
            pk_id=user.id
        )
        self.assertEquals(response.status_code, 201)

Code commits done for this post:

Release:

13-05-17-user-api

Tools and applications used:

Acknowledgements of links I've found usefull while writing this article:

Others:

Django-Rest-Framework:

StackOverFlow:

Accomplished:

Issue #22 - User API

Final word

I wanted to make all of the issues that are there for 1st Milestone. Unfortunatelly I already see that it's impossible to make 6 big features in 4 hours (even less).

I'll move end time till Wednesday and let's see how much progress I will make.

Next time I'll just have to create less optimistic estimation of time per issue :)

That's all, thanks ! Keep in touch :)

Biking Endorphines - User Model and TDD.

2017-05-12T20:00:00+02:00

User Model and the Tdd

So I've got BMI name per specific bmi value. #10 issue.

I've started work on it. Went to master branch. Created feature-branch.

And then.... I've found out django-models that has no test-coverage or other things like that and looks a bit more then little as not TDD-approach.

Well... let's go and fix that, shall we ?

Fixing User Model not tested yet.

So I've figure out I'll first create test for checking if fields exists in model.

Initially I've created test method that was a bit product-specific, but then finally I've moved it to a GenericModelTestCase class and added test only in this generic-type class.

Then in specific-model-class I only inherited by this generic-type class and added cls and fields fields :)

The GenericModelTestCase looks like that (in generics.py module):

"""
All generic type of classes used in django-tests
"""
from django.test import TestCase
from django.contrib.auth.models import User as AuthUser


class GenericModelTestCase(TestCase):
    """
    Generic Model Test Case.
    Contains:
    - assert_fields - assertion for checking if all fields in model exists
    """
    cls = AuthUser
    fields = []

    def assert_fields(self, fields):
        """ asserting if fields exists in model"""
        fields_in_model = []
        # fields as django fields:
        for field in self.cls._meta.get_fields(include_hidden=True):
            if field.name in fields:
                fields_in_model.append(field.name)
        # method as fields:
        for method in self.cls.__dict__:
            if method in fields and method not in fields_in_model:
                fields_in_model.append(method)
        assert sorted(set(fields_in_model).intersection(fields)) == sorted(fields)

    def test_fields(self):
        """ Uses asserting fields for checking with naming convention of django-type """
        self.assert_fields(self.fields)

And the UserModelTestCase

"""
Unit tests for User model
"""
from web.models import User
from web.tests.django_type.generics import GenericModelTestCase


class UserModelTestCase(GenericModelTestCase):
    """ User Model TestCase """
    cls = User
    fields = ['id', 'name', 'surname', 'weight', 'height', 'bmi']

Clean and simple ! :)

Now... how about that feature branch, huh?

Yeah I know I should not mix feature branch with a fixing lack-of-tests branch, but for sake of simplicity I'll skip that this time.

Now moving to the feature it self.

It should be pretty simple, isn't it ?

Let's find it out ! :)

Feature branch code

At models.py, User class :

class User(models.Model):
    """
    User model for obtaining personal information about biking riders
    """
    name = models.CharField(max_length=50)
    surname = models.CharField(max_length=100, default="")
    weight = models.IntegerField(default=0)
    height = models.IntegerField(default=0)

    def __unicode__(self):
        """
        Returns User information when using str/printing
        """
        return self.name

    def bmi(self):
        """
        Body Mass Index calculator simplified to number
        """
        return (self.weight / (self.height * self.height)) * 10000

    def bmi_health_name(self):
        """
        BMI Health Name - Returns proper naming for value of BMI
        """
        if self.bmi() < 0:
            return None
        if self.bmi() >= 0 and self.bmi() <= 18.5:
            return "Underweight"
        if self.bmi() > 18.5 and self.bmi() <= 24.9:
            return "Normal weight"
        if self.bmi() > 24.9 and self.bmi() <= 29.9:
            return "Overweight"
        if self.bmi() > 29.9:
            return "Obesity"

At tests_user.py :

"""
Unit tests for User model
"""
from mock import MagicMock

from web.models import User
from web.tests.django_type.generics import GenericModelTestCase


class UserModelTestCase(GenericModelTestCase):
    """ User Model TestCase """
    cls = User
    fields = ['id', 'name', 'surname', 'weight', 'height', 'bmi']

    def setUp(self):
        """SetUp"""
        self.user = User()

    def assert_bmi_health(self, bmi, expected):
        """ Asserts BMI health name"""
        self.user.bmi = MagicMock(return_value=bmi)
        assert self.user.bmi_health_name() == expected

    def test_bmi_health_name(self):
        """ tests for bmi_health_name method """
        self.assert_bmi_health(-1, None)
        self.assert_bmi_health(0, "Underweight")
        self.assert_bmi_health(1, "Underweight")
        self.assert_bmi_health(18.5, "Underweight")
        self.assert_bmi_health(18.6, "Normal weight")
        self.assert_bmi_health(25.0, "Overweight")
        self.assert_bmi_health(30.0, "Obesity")

Code commits done for this post:

Tools and applications used:

Release

You can find release of source code here.

Acknowledgements of links I've found usefull while writing this article:

DjangoProjects.com:

Model _meta API

StackOverFlow:

Accomplished:

TDD Approach for Django Models!

Final word! -> NEW BADGE UNLOCKED!

You have one new badge unlocked ! - Making Django Generic Model Test Case! - Huuuraaay ! :)

Check my 1st Milestone here

Will I be able to complete it till Sunday?

Give a comment in box below :)

Thanks ! Keep in touch :)

Biking Endorphines - Pull Requests and Merge Conflicts!

2017-05-11T22:00:00+02:00

What are conflicts ?

If you are not familiar with git conflicts, than you were very lucky using git or you didn't use it in such ways that you could fly instead of run.

Conflicts are such a wonderfull things, that sometimes you just can't help yourself by swearing at them (NOT people who did commits - obviously!).

But to be more professional - let's go back and explain - Conflicts... hmmm

Well you have your branch, let's say it's a master branch. You create new branch called "feature/New-Wonderful-Functionality".

Let's say you create 2 commits and one of them makes changes in README.md file in line 2. Then your manager/supervisor/whoever goes to your room/cubic/table and says has a quick hot-fix that needs urgent change in README.md file.

So you create a new branch based on master called "hotfix/Fixing-Bad-Name-in-README". You make changes in README.md file in 2 line. You commit your changes, push changes to remote branch and then make pull-request and merge them finally to master.

And then you forget about it. You go back to your new-wonderful-functionality branch. Make some additional commits (say - 2). You push your branch to remote branch.

Then you want to make pull-request to master. And That's when magic happens.

Your master branch has already changes in README.md at line 2 that are different then what you are having. Git don't know which change has priority and thus fires with "I"m not going to merge your code automatically - your changes conflicts with what you already have in your branch!" :)

So you have to work with this to fix issue. In most cases you will take only elements from your "hotfix" and leave your "new-wonderful-functionality" in next line.

Also check in Acknowledgements links that may or may not explain it better.

What type of conflicts I've encountered ?

Well let's see it in action, shall we ?

After making this recording I also needed to make a pushing with force:

git push origin +HEAD:branch

Notice the +HEAD - sign + makes forcing pushing to remote branch regretless of what was before on that particular branch.

Tools and applications used:

Acknowledgements of links I've found usefull while writing this article:

Other:

A better pull request

StackOverFlow:

How to resolve merge conflicts in Git

Accomplished:

Merging all feature-branches at biking-endorphines

What about you? How do YOU manage your conflicts?

Give me your comment, how do you manage your conflicts? Do you use a GUI tools ? Do you make it via console/command line?

If you reached this point - I personally want to thank you for time you've spend!

Thanks ! Keep in touch :)

Pelican Plugins playground still continue - Asciinema!

2017-05-10T22:00:00+02:00

Asciinema, what is that?

Asciinema is a tty(console session) recorder that enables to copy-paste a recorded commands when watching them played.

Best of all, you can use not only their site, but you can also make your own player at your site more about that here

How to use Asciinema?

Example:

<link rel="stylesheet" type="text/css" href="css/asciinema-player.css" />
<asciinema-player src="asciinema/biking_endorphines_pyreverse.json"></asciinema-player>
<script src="scripts/asciinema-player.js"></script>

Pretty easy huh? Well, yea, you may use then only this:

<asciinema-player src="asciinema/biking_endorphines_pyreverse.json"></asciinema-player>

And that is fine. Now why would I create a plugin for such a thing? To make my life a bit easier, and not having to repeat myself by adding over and over again the css and js files into markdown files or the theme itself (theme then is changeable, I'm not stuck with only this changed theme).

Demo?? Where can I see how it works ??

You could check how it works at my previous post

What are key concepts?

I want to have added javascript and css stylesheets whenever a tag will be found in markdown file. When not, then plugin should not add anything to content of markdown files at the generation stage.

What's now?

Well I have'nt got enough proper time to sit down and make this plugin yet. But I've created pelican-asciinema repository at github.

Tools and applications used:

I'll give feedback next Wednesday :)

Thanks!

Accomplished:

1. Start working on plugin for asciinema player.

What's next

1. See ISSUES task-list for this 'mini' growing project :)

Biking Endorphines - The Plan for Future!

2017-05-07T17:15:00+02:00

The Plan!

Lately I've found that there is only 3 weeks left for BikingEndorphines project to be done for GetNoticed2017 contest.

Here you can check my Get-Noticed profile.

So I'm currently changing my working into a weekly sprints (something similar to a scrum-based development workflow.

After each week, bunch of features will be done. I'll try my best to make as much articles about that features :)

At the end of each week I'll give You feedback, about my progress in list of:

Completed
In-progress
Not-started-yet

So stay tuned! A lot of new content is comimg :)

Details of the Plan!

Since there is only 3 weeks left to at least complete key-features, I presume the best will be to make each week about the same things. This will help me to keep up and stay focused on one stage of project for few days and then go to other stage of project.

Week for Python

Key features to be done:

REST API with :
- Users
- Endorphine-Badges Acquired
- Routes User have been (based on gpx files)
- Possibility to put new routes using endpoint and a gpx input-file.
- API requests per-user.
- Secure REST API with https security on a server-site (i.e. nginx)

Nice to have, if time will be given:

Frontend web-page with:
- log-in page.
- Statistics about each route user have been
- Endorphine Badges Acquired

Week for Android

Key features to be done:

Android UI mockups.
Connection to REST API using some library check at android-arsenal
Starting/Stopping route recording
Sending gpx route to REST API automatically whenever Wifi/Internet is available
Gathering routes history.

Nice to have, if time will be given:

Preferences of sending gpx files (if using wifi or not)
Gathering Badges Acquired information

Week for mashup!

Key features to be done:

Android and Python backend syncing data between each-other
Moving locally accessible rest-service to outside-world i.e. using Heroku.
Problems that for now I'm not fully aware and may raise while working on those 2 weeks above.
Resolving as much as possible of 'nice to have' from past 2 weeks.

Tools and applications used:

Let's go and TDD the whole WORLD ! :D

All features will be done in a Test-Driven-Development workflow

Acknowledgements of links I've found usefull while writing this article:

Other: - Best Practice System for Organizing and Tagging github Issues

StackOverFlow:

Accomplished:

1. Plans for Biking-Endorphines.

What's next

1. Badge Gathering source-code and tests.

Biking Endorphines - How to secure our REST-API?

2017-05-05T21:00:00+02:00

Security - why should I care?

Let's say that you will want to make your REST-API available for others to use it. Let's face it - you will need to restrict use of API based on users. If you don't do that, you may face the reality of having issues with your data that's used by your API.

Let's play old game "What if".

You have your API for User accounts. You don't restrict access. What if someone discover API that you didn't published and reuse it in his own purpose ? i.e. for making dangerous calls that may lead to DATA LOSS ?

You may have some type of recovery mode (backups/snapshots), but what if this problem will not end in an "one-time-shot"?

So let's move to some possibilities to restrict access to your API !

What Django REST API can give out of the box?

In article - Authentication with Django-Rest-Framework you can find information how to add some security by using Authentication there.

There are :

BasicAuthentication
TokenAuthentication

For the purpose of checking out, I'll use the TokenAuthentication even if it's not best secure. But still it's available within rest-framework (out of the box thinking :) )

What are alternatives to django-rest-framework security?

Well there is a lot of frameworks that supports OAuth, but not all will natively support API.

Below you can check some of them that are mentioned at the "Authentication" article at Django-Rest-Framework:

Source code I've applied to Biking Endorphines.

At /bikingendorphines/api/models.py:

from django.db.models.signals import post_save
from django.dispatch import receiver
from rest_framework.authtoken.models import Token
from django.conf import settings

# This code is triggered whenever a new user has been created and saved to the database

@receiver(post_save, sender=settings.AUTH_USER_MODEL)
def create_auth_token(sender, instance=None, created=False, **kwargs):
    if created:
        Token.objects.create(user=instance)

At /bikingendorphines/bikingendorphines/settings.py:

# in `INSTALLED_APPS` this:
'rest_framework.authtoken',

# And added this

REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.IsAuthenticated',
    ),
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
    )
}

All I needed to do is create a new superuser and Grab Token from database, then apply it to requests like this (httpie in example used)

http GET 127.0.0.1:8000/api/user/ 'Authorization: Token XXXXX'

Final result

Check out my release for this article: 5-05-17 secure-api

Code commits done for this post:

Adds TokenAuthentication to project

Tools and applications used:

Acknowledgements

Django-Rest-Framework Token Authentication- at StackOverFlow

Accomplished:

1. How to secure biking-endorphines REST-API.

What's next

1. Plans for Biking-Endorphines

2. Badge Gathering source-code and tests.

FancyBox Plugin Heads Up!

2017-05-03T21:30:00+02:00

FancyBox Pelican Plugin

Yes! It's a ready-to-use product :)

So... how to use it ??

FancyBox Plugin Installation!

I will add a README information in github project, but for now let's go to specifics of HOW TO:

Best is if you use release 1.0 version or code from master, in below example I'll use release:

wget https://github.com/anselmos/pelican-fancybox/archive/v1.0.zip

Then unzip file.

You should have a directory called "v1.0" or other that you have named your zip file.

Change name into fancybox and move it to your pelican-plugins directory

Now add plugin as usually into pelican:

PLUGIN_PATHS = ['pelican-plugins']
PLUGINS = ['fancybox']

That's for only this plugin, but if you have more plugins then this, you just add fancybox or other name of directory inside of which fancybox source code exists, into PLUGINS list.

And voilà. You should be ready to use plugin!

Fancybox Plugin Usage!

But wait ... so how can you use it ??

That's the magic! You only add something like this into article:

Title: First
Date: 2017-04-03
When writing a post or page, add a footnote like this:

test


<fancybox href=http://img06.deviantart.net/63b8/i/2012/178/2/8/jarvis___shield_interface___rainmeter_theme_by_edreyes-d54z5ky.jpg>Url</fancybox>

Final result

Final result is available at master branch of project or at v1.0 release tag

Code commits done for this post:

Tools and applications used:

Little UPDATE :)

I've finished drawing a pseudo logo for my plugin that you can check here. or at Instagram

Hope you'll like it !

Please leave a comment below :)

Thanks!

Accomplished:

1. Make final commit for source-code of fancybox plugin

2. Draw a logo for pelican-fancybox-plugin

What's next

1. Start working on plugin for asciinema player.

2. See ISSUES task-list for this 'mini' growing project :)

Refreshed Blog Look!

2017-04-30T10:30:00+02:00

Refreshed Look!

I've decided that my blog needs a little bit refreshing and adding some cool stuff.

So I've used knowledge from my previous project - anselmos-pelican-blog and used it well (at least I hope I did :) )

You can compare how this blog looked like before change and after in this images:

Before

After

What's next ??

Well... let's see :

1. Pelican-Fancybox plugin that I'm currently working on (finalizing)

2. Usage of Github "Gists" with Pelican-Gist Plugin

3. Other that I might forgot... :)

Thanks!

Do you like the refreshed style of blog ? Give a comment, don't be shy! :)

Stay Tuned! Soon new posts !

Biking Endorphines - REST-API Documentation

2017-04-28T00:00:00+02:00

What can you use for API documentation?

According to topic Documenting your API at Django-Rest-Framework you have following options to use as API documentation:

1. Django REST Framework documentation

This is the most basic documentation that you can get "out of the box" - almost.
The only dependency needed here is the "renderer" - to be more precise - the `coreapi`.

2. DRF Docs

Quoting information Django-REST-Framework Documentation topic:

DRF Docs allows you to document Web APIs made with Django REST Framework and it is authored by Emmanouil Konstantinidis.
It's made to work out of the box and its setup should not take more than a couple of minutes.
Complete documentation can be found on the website while there is also a demo available for people to see what it looks like.
Live API Endpoints allow you to utilize the endpoints from within the documentation in a neat way.

3. Django REST Swagger

Quoting information Django-REST-Framework Documentation topic:

Marc Gibbons' Django REST Swagger integrates REST framework with the Swagger API documentation tool. The package produces well presented API documentation, and includes interactive tools for testing API endpoints.
Django REST Swagger supports REST framework versions 2.3 and above.
Mark is also the author of the REST Framework Docs package which offers clean, simple autogenerated documentation for your API but is deprecated and has moved to Django REST Swagger.
Both this package and DRF docs are fully documented, well supported, and come highly recommended.

4. DRF AutoDocs

Quoting information Django-REST-Framework Documentation topic:

Oleksander Mashianovs' DRF Auto Docs automated api renderer.
Collects almost all the code you written into documentation effortlessly.
Supports:

- functional view docs
- tree-like structure
- Docstrings:
- markdown
- preserve space & newlines
- formatting with nice syntax
- Fields:
- choices rendering
- help_text (to specify SerializerMethodField output, etc)
- smart read_only/required rendering
- Endpoint properties:
- filter_backends
- authentication_classes
- permission_classes
- extra url params(GET params)

Why we chose DRF for documentation of Biking-Endorphines

Other options ( DRF Docs and Django REST Swagger) uses swagger-ui for documentation. In my professional career I've already used swagger-ui.

I'd probably do some follow-up about how to use other options, but this time I want to use something out-of-box.

How do we implement documentation of our project??

Just like in other's options, first you need to document your API! So let's check how it will look like, shall we?

class UserList(generics.ListCreateAPIView):
    """
    get:
    Return a list of all existing users.

    post:
    Create a new user instance.
    """
    queryset = User.objects.all()
    serializer_class = UserSerializer

Now that we finally add documentation (it was just few comment-code on class ), we can add dependencies to our django application:

(At urls.py):

from rest_framework.documentation import include_docs_urls

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^api/user/$', views.UserList.as_view()),
    url(r'^docs/', include_docs_urls(title='API Documentation')), # TEST
]

Also for API to be accessible and handled you need to install coreapi that's used as default for rendering.

pip install coreapi
pip install coreapi-cli

Final result

I've figure out best solution to deliver code that is not yet merged into master is by using Annotated tagging.

So to get source-code you can wget'it or check it at this tag-release information

Code commits done for this post:

Tools and applications used:

LL (Lessons Learned)

1. Heroku - how things failed.

I had too-ambitious plan of deploying source-code to Heroku to have a working example od Documentation out-of-the-box.

Unfortunatelly last time I've used Heroku was in January 2016, so I didn't remember a lot about it.

Also proper setup of project was needed from beginning, moving on with tutorial step-by-step.

As our project is not best compatible with heroku setup, I've found myself in situation where there is more work needed to properly deploy content.

Everything would work, but unfortunatelly the static pages setup was not good enought for documentation, which you can check here

Solution:

Follow tutorial about How to use Heroku in Django-Based projects. For those impatient, you can use template for Django 1.10 that is supported with Heroku

I will make a follow-up article about how to properly setup your project using Heroku, and also how to fix your project if have different directory structure (as BikingEndorphines has).

Tips & Tricks

1. Too Many Ancestors! Pylint Error.

So After using documentation and trying the generic-class based Django REST Framework I've found myself having this issue.

Initialy I've made a workaround using # pylint disable=too-many-ancestors, but then I've figure out - It's ugly, it should be setup in pylintrc!

And so the hunt for setup in pylintrc began, finally checking this article at StackOverFlow I've found resolution - by changing one variable at pylintrc - max-parents=15 which will conclude in using more then 7 default class inheritance.

Acknowledgements

CookieCutter-Django-Rest

Accomplished:

1. Making API documentation - research for best API documentation!

What's next

1. How to secure biking-endorphines REST-API.

2. Badge Gathering source-code and tests.

FancyBox Plugin Finalizing

2017-04-26T23:50:00+02:00

FancyBox Pelican Plugin - are you done already?

The FancyBox is almost complete. Need to adjust some elements. But main functionality is almost complete.

Down below are all functions that I've created:

Replace function.

Needed to create a replace function for replacing fancybox markdown element from article with a fancybox-type element used with fancybox javascript engine.

Test for replace function:

def assert_replace(article, expected):
    "Asserts replace equals expected"
    assert replace(article) == expected

def test_given_article_with_fancybox_replace():
    "Checks if Replace method makes replacement of <fancybox></fancybox>element into <a class='fancybox_group'></a> "

    article = Article('Data data data\n <{}>TEST</{}>\ndata data data'.format(FANCYBOXNAME, FANCYBOXNAME))
    expected = 'Data data data\n <a class="{}">TEST</a>\ndata data data'.format(CLASS_SELECTOR)
    assert_replace(article, expected)

def test_given_multiple_fancybox_elements_in_article_replace():
    "Checks if multiple fancybox elements will be replaced with fancybox-type engine element"

    article = Article('Data data data\n <{}>TEST</{}>\ndata <{}>TEST</{}>data <{}>TEST</{}>data'.format(FANCYBOXNAME, FANCYBOXNAME, FANCYBOXNAME, FANCYBOXNAME, FANCYBOXNAME, FANCYBOXNAME))

    expected = 'Data data data\n <a class="{}">TEST</a>\ndata <a class="{}">TEST</a>data <a class="{}">TEST</a>data'.format(CLASS_SELECTOR, CLASS_SELECTOR, CLASS_SELECTOR)

    assert_replace(article, expected)

Source code:

def replace(article):
    "Replaces fancybox tag with <a class='fancybox'></a>"

    elements_fancybox, soup = find_fancybox_element(article)
    for fancybox in elements_fancybox:
        fancybox.name = TAG_REPLACEMENT
        fancybox['class'] = CLASS_SELECTOR
        fancybox['href'] = fancybox.text
        fancybox.append( BeautifulSoup("", "html.parser").new_tag("img", href=fancybox.string) )
    return str(soup)

Add Dependency function.

"add_dependency" function - adds all javascript and css dependency files as / to source of markdown article. Probably now it's not added, but soon this "dependency" will only be added if element is found in markdown article source file.

Test for "add dependency" function:

def test_given_article_add_dependency():
    "Checks if article contains css element after using 'add_css' function - only if fancybox element exists"

    expected_content = '<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4/jquery.min.js" type="text/javascript"></script><script src="fancybox/jquery.fancybox-1.3.4.pack.js" type="text/javascript"></script>'
    expected_content += '<link href="fancybox/jquery.fancybox-1.3.4.css" media="screen" rel="stylesheet" type="text/css"/>'

    article_content = 'Data data data\n <{}>TEST</{}>\ndata data data'.format(FANCYBOXNAME, FANCYBOXNAME)
    article = Article(article_content)

    expected = Article(expected_content + article_content)

    assert str(add_dependency(article)._content) == str(expected._content)

Source code:

def add_dependency(article):
    'Adds CSS/JS dependency to article only if article contains fancybox element'
    script_tag = BeautifulSoup("", "html.parser").new_tag("script", type="text/javascript", src="")
    css_tag = BeautifulSoup("", "html.parser").new_tag("link", rel="stylesheet", type="text/css", href="", media="screen")
    script_tag['src'] = DEPS_JS_JQUERY_URL

    content = str(script_tag)

    script_tag['src'] = DEPS_JS_FANCYBOX_URL
    content += str(script_tag)

    css_tag['href'] = DEPS_CSS_FANXYBOX_URL
    content += str(css_tag)

    article._content = content + article._content
    return article

Add Binding Fancyboxscript function.

This function adds Javascript binding between fancybox elements in markdown source and fancybox engine.

Tests :

def test_given_article_add_binding_fancyboxscript():
    "Checks if article contains javascript binding between name of class and fancybox script"
    article_content = 'Data data data\n <{}>TEST</{}>\ndata data data'.format(FANCYBOXNAME, FANCYBOXNAME)
    expected_content = article_content
    expected_content += "<script>"
    expected_content += """$(document).ready(function() {$("a.fancybox").fancybox({'hideOnContentClick': true});});"""
    expected_content += "</script>"

    article = Article(article_content)
    expected = Article(expected_content)
    assert str(add_binding_fancyboxscript(article)._content) == str(expected._content)

Source code:

def add_binding_fancyboxscript(article):
    "Adds biding for fancybox script with class selector"
    binding = "<script>"
    binding += JS_BIDING_CONTENT
    binding += "</script>"
    article._content = article._content + binding
    return article

Source code of plugin-itself

def fancybox_plugin(generator):
    "Fanxybox plugin - temporary code placement"
    for article in generator.articles:
        article._content = add_dependency(article)._content
        article._content = add_binding_fancyboxscript(article)._content
        article._content = replace(article)

What needs to be done (a follow-up article about that :) )

In short - functions needs to be change into methods, links to js/css needs to be adjusted. Some additional tests needs to be added.

Final result

Final result can be checked at this branch

Code commits done for this post:

Tools and applications used:

Accomplished:

1. Make ALMOST final commits for source-code of fancybox plugin

What's next

1. Make final commit for source-code of fancybox plugin

2. Start working on plugin for asciinema player.

3. See ISSUES task-list for this 'mini' growing project :)

Biking Endorphines - REST-API within Django

2017-04-26T00:00:00+02:00

What frameworks exists that supports REST-API within python ?

1. Django REST Framework

Quoting information from site:

Some reasons you might want to use REST framework:

- The Web browsable API is a huge usability win for your developers.
- Authentication policies including packages for OAuth1a and OAuth2.
- Serialization that supports both ORM and non-ORM data sources.
- Customizable all the way down - just use regular function-based views if you don't need the more powerful features.
- Extensive documentation, and great community support.
- Used and trusted by internationally recognised companies including Mozilla, Red Hat, Heroku, and Eventbrite.

2. Flask and Bottle

Quoting information from site about Bottle:

Bottle is a fast, simple and lightweight WSGI micro web-framework for Python. It is distributed as a single file module and has no dependencies other than the Python Standard Library.

- Routing: Requests to function-call mapping with support for clean and dynamic URLs.
- Templates: Fast and pythonic built-in template engine and support for mako, jinja2 and cheetah templates.
- Utilities: Convenient access to form data, file uploads, cookies, headers and other HTTP-related metadata.
- Server: Built-in HTTP development server and support for paste, fapws3, bjoern, gae, cherrypy or any other WSGI capable HTTP server.

3. Flask-restful

Quoting information from site :

Flask-RESTful is an extension for Flask that adds support for quickly building REST APIs. 
It is a lightweight abstraction that works with your existing ORM/libraries. 
Flask-RESTful encourages best practices with minimal setup. 
If you are familiar with Flask, Flask-RESTful should be easy to pick up.

4. Python-Eve

Quoting information from site :

Eve is an open source Python REST API framework designed for human beings.
It allows to effortlessly build and deploy highly customizable, fully featured RESTful Web Services.

Eve is powered by Flask, Cerberus, Events and MongoDB.
Support for SQL-Alchemy, Elasticsearch and Neo4js as alternate backends is provided by community extensions.

The codebase is thoroughly tested under Python 2.6, 2.7, 3.3, 3.4, 3.5, 3.6 and PyPy.

5. Falcon

Quoting information from site :

Falcon is a very fast, very minimal Python web framework for building microservices, app backends, and higher-level frameworks.
It is used by a growing number of organizations around the world, including Cronitor, EMC, Hurricane Electric, OpenStack, Opera Software, Wargaming, and Rackspace.

The Falcon web framework encourages the REST architectural style, meaning (among other things) that you think in terms of resources and state transitions, which map to HTTP methods.

A number of Falcon add-ons, templates, and complimentary packages are available for use in your projects.
We've listed several of these on the Falcon wiki as a starting point, but you may also wish to search PyPI for additional resources.

Falcon can be deployed in a variety of ways, depending on your needs.
The framework speaks WSGI, and works great with CPython 2.6 and 2.7, PyPy, and CPython 3.3+.

6. Cornice

Quoting information from site :

Cornice provides helpers to build & document REST-ish Web Services with Pyramid, with decent default behaviors.
It takes care of following the HTTP specification in an automated way where possible.

We designed and implemented cornice in a really simple way, so it is easy to use and you can get started in a matter of minutes.

7. Tastypie

Quoting information from site :

Tastypie is a webservice API framework for Django.
It provides a convenient, yet powerful and highly customizable, abstraction for creating REST-style interfaces.

8. Others maybe included here

What framework our Biking-Endorphines will use?

1. My selection criteria:

1.A) REST API creation
1.B) TDD support
1.C) Different type of data formats i.e. json and xml (best - out of the box)
1.D) Documentation (best - out of the box)

2. What framework looks like supports all my criteria out of the box?

Well that's not a surprise - the Django Rest Framwork passes all my criteria.

How to use Django Rest Framework?

1. Serializers.

First you need to have any type of Serializer for your objects.

In our projects example I'll use the "User" class definition for creating serializer:

class UserSerializer(serializers.Serializer):
    """
    User serializer
    """
    #pylint: disable=invalid-name
    id = serializers.IntegerField(read_only=True)
    name = serializers.CharField(required=False, allow_blank=True, max_length=50)
    surname = serializers.CharField(required=False, allow_blank=True, max_length=100)
    weight = serializers.IntegerField(required=False, default=0)
    height = serializers.IntegerField(required=False, default=0)

    def create(self, validated_data):
        """
        Create and return a new `User` instance, given the validated data.
        """
        return User.objects.create(**validated_data)

    def update(self, instance, validated_data):
        """
        Update and return an existing `User` instance, given the validated data.
        """
        instance.name = validated_data.get('name', instance.name)
        instance.surname = validated_data.get('surname', instance.surname)
        instance.weight = validated_data.get('weight', instance.weight)
        instance.height = validated_data.get('height', instance.height)
        instance.save()
        return instance

But wait.... what about tests ? - Good point.

"""
Tests for API
"""
import unittest
from api.serializers import UserSerializer
from web.models import User



class TestUserSerializer(unittest.TestCase):
    "UserSerializer tests"

    def test_user_serialize_to_json(self):
        "test if serializing to JSON works"

        mocked = User()
        mocked.name = "Anselmos"
        mocked.surname = "Somlesna"
        mocked.weight = 80
        mocked.height = 175

        user_serialized = UserSerializer(mocked)
        self.assertEqual(
            (user_serialized.data),
            {'height': 175, 'surname': u'Somlesna', 'id': None, 'weight': 80, 'name': u'Anselmos'}
        )

2. Views for API.

Now lets create a View for our Serializer and then add it to our urls.py

"""
Views
"""
from rest_framework.views import APIView
from rest_framework.response import Response

from api.serializers import UserSerializer
from web.models import User


class UserList(APIView):
    """
    List all User's, or create a new snippet
    """

    def get(self, request, format=None):
        "GET"
        users = User.objects.all()
        serializer = UserSerializer(users, many=True)
        return Response(serializer.data)

Test:(yeah it's not enough for a test, but still simplest one:)

class TestUserList(unittest.TestCase):
    "UserList tests"

    def test_user_get_return_json(self):
        "test if using get returns json data"

        view = UserList.as_view()
        factory = APIRequestFactory()
        request = factory.get("/api/user/")
        force_authenticate(request)
        response = view(request)
        self.assertEquals(response.status_code, 200)

3. Urls for proper handling on outside:

In urls.py

    url(r'^api/user/$', views.UserList.as_view()),

Final result

Final result is available at this branch

Code commits done for this post:

Tools and applications used:

Acknowledgements

Accomplished:

1. REST-API for Test-Database.

What's next

1. Making API documentation - research for best API documentation!

2. How to secure biking-endorphines REST-API.

3. Badge Gathering source-code and tests.

Biking Endorphines - Test Database and REST API

2017-04-22T23:50:00+02:00

Prerequisites

1. REST API vs SOAP API

So you are here to know what are the difference between API webservices?

Let's go with simple what are cons/pros of each solution!

1.A REST API Pros and Cons

PROS:

Lower amount of data needed to deliver actual content
Available two types of data formats (at least that's what I know of : JSON and XML)
Thanks to usage of JSON data format - simplified access to data within i.e. javascript/Android Applications etc.
Supports CRUD operations (Create, Read, Update, Delete) - that are correlated with HTTP convension (Create -> PUT/POST, Read -> Get, Update -> PUT/POST/PATCH, Delete-> DELETE)

CONS:

REST API formally does not have any type of WSDL and therefore you don't have any formal endpoints definition (only as description i.e. using other services)

1.B SOAP API Pros and Cons

PROS:

Contains formal scheme defining endpoints within WSDL
Output of endpoints data is formalized in WSDL and therefore you can check for expected output from that endpoints.

CONS:

Only XML with WSDL scheme
Lack or small amount of support for SOAP-XML at Android Community
Large amount of data to deliver actual content

2. Why do you want to use REST API ?

REST API is used from Client's perspective rather for faster data access with low amount of data delivered.

In my humble opinion that's why Android Community does not give such great support for SOAP.

Also that's your answer: If you are planning to use this webservice as an Endpoint for other Clients (Mobile Device, other web-applications i.e. based on javascript engine), than REST API is webservice for you!

But If you don't mind more amount of data and a little bit more complex data binding (parsing/sending/reading) but better formalization of data, than you obviously choose SOAP API.

One of the most important elements that makes SOAP API great is it's WSDL which helps to correlate how endpoints looks like, what data ought to be inside of them - which in later phase - helps you to more properly bind data and be sure that only this type of data should come out them.

Initial Database for REST API in Biking Endorphines project.

1. Test Database assumptions.

What I love about django is it's ORM(Object-Relational-Mapping). It offers you usage of old-school traditional database-in-file SQLite, but also other Database engines like i.e. Postgres.

You can easily switch between them, only thing that needs to be moved is data itself. But since it's a ORM you can manipulate database and create Test-data.

For our Biking Endorphines project, expected is to create tables with data like below:

User - just usual stuff like:

(some already exists as part of bmi-checker functionality.)
- id,
- name,
- surname,
- weight,
- height
Biking Route with Points :

Route:
- id,
- route_name,(for easier search)
- avg_route (predefined avg for this route, not related to user
User-Route:
- id_user,
- id_route
Point:
- id,
- id_route,
- point, (lat, long)
- elevation,

2. Type of database.

For now I'll use SQLite - a default database delivered with django instance.

I will probably move to my favourite database - PostgreSQL in future.

There is also a plan to create article about PostgreSQL in docker - So keep in touch :)

3. Database scheme.

So I've created a class-based ORM scheme within django. Here is how it looks like after python manage.py dbshell >.schema :

CREATE TABLE "django_migrations" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "app" varchar(255) NOT NULL, "name" varchar(255) NOT NULL, "applied" datetime NOT NULL);
CREATE TABLE "auth_group" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "name" varchar(80) NOT NULL UNIQUE);
CREATE TABLE "auth_group_permissions" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "group_id" integer NOT NULL REFERENCES "auth_group" ("id"), "permission_id" integer NOT NULL REFERENCES "auth_permission" ("id"));
CREATE TABLE "auth_user_groups" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "user_id" integer NOT NULL REFERENCES "auth_user" ("id"), "group_id" integer NOT NULL REFERENCES "auth_group" ("id"));
CREATE TABLE "auth_user_user_permissions" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "user_id" integer NOT NULL REFERENCES "auth_user" ("id"), "permission_id" integer NOT NULL REFERENCES "auth_permission" ("id"));
CREATE UNIQUE INDEX "auth_group_permissions_group_id_0cd325b0_uniq" ON "auth_group_permissions" ("group_id", "permission_id");
CREATE INDEX "auth_group_permissions_0e939a4f" ON "auth_group_permissions" ("group_id");
CREATE INDEX "auth_group_permissions_8373b171" ON "auth_group_permissions" ("permission_id");
CREATE UNIQUE INDEX "auth_user_groups_user_id_94350c0c_uniq" ON "auth_user_groups" ("user_id", "group_id");
CREATE INDEX "auth_user_groups_e8701ad4" ON "auth_user_groups" ("user_id");
CREATE INDEX "auth_user_groups_0e939a4f" ON "auth_user_groups" ("group_id");
CREATE UNIQUE INDEX "auth_user_user_permissions_user_id_14a6b632_uniq" ON "auth_user_user_permissions" ("user_id", "permission_id");
CREATE INDEX "auth_user_user_permissions_e8701ad4" ON "auth_user_user_permissions" ("user_id");
CREATE INDEX "auth_user_user_permissions_8373b171" ON "auth_user_user_permissions" ("permission_id");
CREATE TABLE "django_admin_log" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "object_id" text NULL, "object_repr" varchar(200) NOT NULL, "action_flag" smallint unsigned NOT NULL, "change_message" text NOT NULL, "content_type_id" integer NULL REFERENCES "django_content_type" ("id"), "user_id" integer NOT NULL REFERENCES "auth_user" ("id"), "action_time" datetime NOT NULL);
CREATE INDEX "django_admin_log_417f1b1c" ON "django_admin_log" ("content_type_id");
CREATE INDEX "django_admin_log_e8701ad4" ON "django_admin_log" ("user_id");
CREATE TABLE "django_content_type" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "app_label" varchar(100) NOT NULL, "model" varchar(100) NOT NULL);
CREATE UNIQUE INDEX "django_content_type_app_label_76bd3d3b_uniq" ON "django_content_type" ("app_label", "model");
CREATE TABLE "auth_permission" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "content_type_id" integer NOT NULL REFERENCES "django_content_type" ("id"), "codename" varchar(100) NOT NULL, "name" varchar(255) NOT NULL);
CREATE UNIQUE INDEX "auth_permission_content_type_id_01ab375a_uniq" ON "auth_permission" ("content_type_id", "codename");
CREATE INDEX "auth_permission_417f1b1c" ON "auth_permission" ("content_type_id");
CREATE TABLE "auth_user" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "password" varchar(128) NOT NULL, "last_login" datetime NULL, "is_superuser" bool NOT NULL, "first_name" varchar(30) NOT NULL, "last_name" varchar(30) NOT NULL, "email" varchar(254) NOT NULL, "is_staff" bool NOT NULL, "is_active" bool NOT NULL, "date_joined" datetime NOT NULL, "username" varchar(150) NOT NULL UNIQUE);
CREATE TABLE "django_session" ("session_key" varchar(40) NOT NULL PRIMARY KEY, "session_data" text NOT NULL, "expire_date" datetime NOT NULL);
CREATE INDEX "django_session_de54fa62" ON "django_session" ("expire_date");
CREATE TABLE "web_point" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "lat" real NOT NULL, "lon" real NOT NULL, "elevation" real NOT NULL);
CREATE TABLE "web_route" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "route_name" varchar(100) NOT NULL, "avg_route" real NOT NULL);
CREATE TABLE "web_routepoint" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "id_point_id" integer NOT NULL REFERENCES "web_point" ("id"), "id_route_id" integer NOT NULL REFERENCES "web_route" ("id"));
CREATE TABLE "web_user" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "name" varchar(50) NOT NULL, "height" integer NOT NULL, "weight" integer NOT NULL, "surname" varchar(100) NOT NULL);
CREATE INDEX "web_routepoint_aa34cbab" ON "web_routepoint" ("id_point_id");
CREATE INDEX "web_routepoint_c5012f0d" ON "web_routepoint" ("id_route_id");
CREATE TABLE "web_userroute" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "id_route_id" integer NOT NULL REFERENCES "web_route" ("id"), "id_user_id" integer NOT NULL REFERENCES "web_user" ("id"));
CREATE INDEX "web_userroute_c5012f0d" ON "web_userroute" ("id_route_id");
CREATE INDEX "web_userroute_3002e0f2" ON "web_userroute" ("id_user_id");

Initial plans for REST API for Biking Endorphines.

I'm planning to create REST-API. Below you can see what 2 endpoints will be created:

1. Gather user data with /user/get

2. Gather Biking routes and points with /route/{id} and /point/{id}

Final result

class User(models.Model):
    """
    User model for obtaining personal information about biking riders
    """
    name = models.CharField(max_length=50)
    surname = models.CharField(max_length=100, default="")
    weight = models.IntegerField(default=0)
    height = models.IntegerField(default=0)

    def __unicode__(self):
        """
        Returns User information when using str/printing
        """
        return self.name

    def bmi(self):
        """
        Body Mass Index calculator simplified to number
        """
        return (self.weight / (self.height * self.height)) * 10000


class Route(models.Model):
    """
    Route model. Defines Route by name, avg_route.
    """
    route_name = models.CharField(max_length=100)
    avg_route = models.FloatField(default=0.0)

    def __unicode__(self):
        "Returns Route name "
        return self.route_name


class UserRoute(models.Model):
    """
    User's routes
    """
    id_user = models.ForeignKey(User, on_delete=models.CASCADE)
    id_route = models.ForeignKey(Route, on_delete=models.CASCADE)

    def __unicode__(self):
        "Returns User id"
        return self.id_user


class Point(models.Model):
    """
    Points similar to gpx-data
    """
    lat = models.FloatField(default=0.0)
    lon = models.FloatField(default=0.0)
    elevation = models.FloatField(default=0.0)

    def __unicode__(self):
        "Returns Point unicode"
        return str(self.lat) + "" + str(self.lon)


class RoutePoint(models.Model):
    """
    Route Points relations
    """
    id_route = models.ForeignKey(Route, on_delete=models.CASCADE)
    id_point = models.ForeignKey(Point, on_delete=models.CASCADE)

    def __unicode__(self):
        "Returns Route-Point relationship unicode"
        return self.id_route

Code commits done for this post:

Adds Test-Database models for REST-API schema.

Tools and applications used:

Acknowledgements

Accomplished:

1. Python backend - initial REST-API.

What's next

1. REST-API for Test-Database.

2. Making API documentation - research for best API documentation!

3. How to secure biking-endorphines REST-API.

4. Badge Gathering source-code and tests.

FancyBox Plugin Finding Fancybox elements in articles

2017-04-19T21:30:00+02:00

Introduction

Lately I've started working on this pelican-plugin that will use special command in markdown article to "magically" change it's content into a fancybox-alike.

I've started it with badly without TDD and that approach failed to leave me with successful plugin created.

My failure was also in copy-pasting others plugins content and trying to re-use it for my own purpose, but it didn't work properly.

This time using TDD and slower movement of source-code I'm working on describing and finding elements of "fancybox" in article.

So no more chit-chat, let's move to the actual source-code and content of this article! :)

Finding Fancybox elements

1. Describing fancybox elements

So I've been thinking how to easily add fancybox element in markdown so it will be easily added to markdown and then also easily readed at plugin for further manipulation of article content.

I've figure I'll start with something, that may be good for beginning and then If I find it will not suit my needs , I'll change it to something else.

That's why for sake of simplicity I'm going to use for now this element:

<fancybox>URL</fancybox>

Let's check how it will work for now! :)

2. Creating testing markdown article and adding it to tests.

I've created an example Article instance with "content" that had the <fancybox>URL</fancybox> element inside. Also I've figure out that I should also check if using pelican it-self will be of great benefit, so I've added a Markdown file in content for pelican.

And sure thing - it did! I found that my plugin function that was registered didn't have proper arguments.

Also I moved finally source-code of my plugin to fancybox.py instead of test_fancybox.py.

3. Adding testing article with fancybox elements

Test function looks like below:

def test_given_article_with_fancybox_find_fancybox_element():
    "Checks for finding fancybox element in article"
    article = Article('Data data data\n <fancybox>TEST</fancybox>\ndata data data')
    assert find_fancybox_element(article)

4. Testing and creating source code.

Source code for now looks like this:

from pelican import signals

class Article(object):
    "A simple Article class"
    def __init__(self, content = ""):
        self.content = content

def find_fancybox_element(article):
    found = None
    for content_line in article.content.split("\n"):
        if "<fancybox>" in content_line:
            found = content_line
    return found

def fancybox_plugin(generator):
    "Fanxybox plugin - temporary code placement"
    print find_fancybox_element(generator.articles[0])

def register():
    "Registers plugin"
    signals.article_generator_finalized.connect(fancybox_plugin)

And yes, I know that I'm currently using checking for first element and it may raise with IndexError.

I'll fix this properly - for sure.

Final result for testing fancybox plugin

"""
FancyBox unittests
"""
from pelican import signals
from fancybox import register
from fancybox import Article
from fancybox import find_fancybox_element
from fancybox import fancybox_plugin

def mock_article_generator():
    "Article generator"
    yield Article()

def assert_receiver_registered(receiver_name):
    'Checks if receiver is registered to signals generator'
    assert signals.article_generator_finalized.has_receivers_for(receiver_name)

def test_plugin_registers():
    """
    Check if plugin registers to article_generator_finalized
    """
    register()
    assert_receiver_registered(fancybox_plugin)

def test_article_generator_return_article():
    """
    Checks if generator return article
    """
    assert isinstance(mock_article_generator().next(), Article)

def test_given_article_generator_check_article_content_exists():
    "Checks if article content field exists in article"
    for article in mock_article_generator():
        assert hasattr(article, 'content')

def test_given_article_with_fancybox_find_fancybox_element():
    "Checks for finding fancybox element in article"
    article = Article('Data data data\n <fancybox>TEST</fancybox>\ndata data data')
    assert find_fancybox_element(article)

Code commits done for this post:

Tools and applications used:

Accomplished:

1. Make more tests and source-code of fancybox plugin

What's next

1. Make final commit for source-code of fancybox plugin

2. Start working on plugin for asciinema player.

3. See ISSUES task-list for this 'mini' growing project :)

GPXReader - initial Endorphine Algorithm

2017-04-19T11:00:00+02:00

Prerequisites

I've initially looked at my source code and Github and found that I've got few things a bit wrongly implemented (not using properly DRY principle.)

I've found also that gpxreader_test branch should be finally merged to master.

I've found also that android-initial startup as module that as not been yet merged to master.

That why initially I needed to make prerequisites like:

1. Merging `gpxreader_test` branch to master

2. Merging `androidapp_submodule` branch to master

3. Fixing problem breaking DRY principle.

As you may check in this commit. Unfortunatelly this fix was in-between working with endorphine-algorithm. My only excuse is that I've followed TDD 3 principles - Make failing test, fix code for test, refactor - to be more precise I've used the "refactor" part.

4. Fixing problem with not using properly unittest module

Instead of using self.assert* I've been using old-school pytest assert. Thanks to this commit I've finally get rid of this old-school assertions.

First Endorphine Algorithm with TDD in mind.

1. Algorithm itself.

Lowest Point You've Been algorithm is all about finding the lowest point in the gpx track that User have been.

If gpx track does not have lowest point, does not have points or those points are of the same elevation, then method should return None.

More comprehensive documentation is in the method documentation.

2. Test Flow for Endorphine-Algorithm for gathering 'Lowest Point You've Been'

So I've been using the TDD for this article. It was a very successful session. I've tried to practice TDD with best intentions and motivation I've found in "Uncle Bob" video about TDD.

First I've started with a failing tests, then created source-code and then refactored code for better quality and readability.

Initially I've started with this code:

    @parameterized.expand([
        ('web/tests/example_data/fast_example_empty_points.gpx'),
    ])
    def test_get_lowest_elevation(self, gpxfile=DEFAULT_GPXFILE_SAMPLE):
        """
        Tests for get_lowest_elevation
        """
        gpxreader = GPXReader(gpxfile)
        if len(gpxreader.get_elevations()) <= 0:
            self.assertEqual(gpxreader.get_lowest_elevation(), None)

Then I've added file called fast_example_empty_points.gpx without any points in it -based on fast_example_trackpoints.gpx file.

After that I've added source code that fixed failing code with :

    def get_lowest_elevation(self):

        if len(self.get_elevations()) <= 0:
            return None

Added some small refactoring and repeated process. I didn't do all the commits for the flow, but I tried to make them as much as I could.

Final result

1. Tests.

    def assert_lowest_elevation_equals(self, expected):
        " Assertion for Equality of get_lowest_elevation with Expected"
        self.assertEqual(self.gpxreader.get_lowest_elevation(), expected)

    #pylint: disable=invalid-name
    @parameterized.expand([
        (DEFAULT_TESTS_EXAMPLES_PATH+'fast_example_empty_points.gpx'),
        (DEFAULT_TESTS_EXAMPLES_PATH+'fast_example_points_no_elevation.gpx'),
    ])
    def test_given_none_or_empty_elevation_return_none(self, gpxfile=DEFAULT_GPXFILE_SAMPLE):
        """
        Tests for get_lowest_elevation
        """
        self.gpxreader = GPXReader(gpxfile)
        self.assert_lowest_elevation_equals(None)

    def test_given_one_available_elevation_return_none(self):
        """
        Tests for get_lowest_elevation
        """
        self.gpxreader = GPXReader(
            DEFAULT_TESTS_EXAMPLES_PATH + \
            "get_lowest_elevation_one_elevation_available.gpx"
        )
        self.assert_lowest_elevation_equals(None)

    def test_given_two_different_points_return_lowest(self):
        """
        Tests for get_lowest_elevation
        """
        self.gpxreader = GPXReader(
            DEFAULT_TESTS_EXAMPLES_PATH + \
            "get_lowest_elevation_two_different_points.gpx"
        )
        self.assert_lowest_elevation_equals(102.0)

    def test_given_two_the_same_points_return_none(self):
        """
        Tests for get_lowest_elevation
        """
        self.gpxreader = GPXReader(
            DEFAULT_TESTS_EXAMPLES_PATH + \
            "get_lowest_elevation_two_the_same_points.gpx"
        )
        self.assert_lowest_elevation_equals(None)

    def test_given_more_then_two_the_same_points_return_none(self):
        """
        Tests for get_lowest_elevation
        """
        self.gpxreader = GPXReader(
            DEFAULT_TESTS_EXAMPLES_PATH + \
            "get_lowest_elevation_more_then_two_the_same_points.gpx"
        )
        self.assert_lowest_elevation_equals(None)

    def test_given_none_and_diff_points_return_minimal_value(self):
        """
        Tests for get_lowest_elevation
        """
        self.gpxreader = GPXReader(
            DEFAULT_TESTS_EXAMPLES_PATH + \
            "get_lowest_elevation_none_and_diff_points_return_minimal_value.gpx"
        )
        self.assert_lowest_elevation_equals(105.0)

2. Source-code of method.

    def get_lowest_elevation(self):
        group_elevation = {}
        for elevation in self.get_elevations():
            if elevation:
                group_elevation[elevation] = elevation

        if len(group_elevation.keys()) == 0:
            return
        if len(group_elevation.keys()) == 1:
            return
        return min(group_elevation.keys())

Code commits done for this post:

Tools and applications used:

LL (Lessons Learned)

1. Deploy_gh_pages travis job fails on "cannot stat.."

Issue that I've found was about travis not deploying pyreverse GitHub Page that should automatically be updated.

What I've found out, was a silly problem with make-tasks that I've recently changed for gpxreader_test branch.

Today I've finally cleaned-up this branch and merged(squashed) it to master. Then I've found out that gh_pages travis task is not working properly.

The issue itself was a bit cryptic at first hand:

cp: cannot stat `../../generated_pyreverse/*': No such file or directory

You can check this issue at Travis for this project

Solution:

I've initially thought that I forgot about adding directory of name generated_pyreverse, but no. Thanks to simplicity of this tool and make-task I needed only few seconds to find out why this happen and what should be done.

I needed to change name of the make task used in travis to "MAKE_GH_PAGE_TASK=deploy_gh_pages_all" instead of "MAKE_GH_PAGE_TASK=deploy_gh_pages"

A kindly reminder for all users of Travis - be sure that if you have any "automatically" deployment, you do test it before running. And all changes in your deployment environment are checked before deploying :)

So , is it working now you may ask ? You may check if you ask :)

Tips & Tricks

1. Move all issues from #TODO in source code to github issues.

At first glance I've used github issues to make all "new" features/issues. Then my habits made their point and I've started using the "TODO" comments in code.

It's not a bad thing, but since I can use github issues I've found it will be useless and makes me create additional #pylint: disable=fixme line that just hides information in code instead of showing it to me.

So I'm moving to github issues. Please if you find any type of #TODO in my code - let me know :)

Accomplished:

1. Add test-cases for business logic to GPXReader - endorphines-algorithms :)

2. Add Endorphine algorithm for 'Lowest Point You Have Been'

What's next

1. Python backend - initial REST-API.

2. Making API documentation - research for best API documentation!

3. Badge Gathering source-code and tests.

GPXReader - Trackpoints Routepoints Waypoints

2017-04-16T12:30:00+02:00

Trackpoints Routepoints and Waypoints

Before today's change, in my source code for tests I had only trackpoints. This worked pretty well, but then I've figure out, I need to be able to check if someone's going to use different type of *points. And obviously I added this to my steps-to-make.

I've created different type of gpx file encodings(TrackPoints/WayPoints/RoutePoints) for tests.

I've added Routepoints and Waypoints. And this made to a tests failure once I've added them to test as parameters.

As I've figure out before, one of the "content" type had different encoding of data and I needed to fix few things in source code.

Finally code has 3 types of different *points.

Final result

Here is final result of test-source-code:

class BaseGPXReaderTest(unittest.TestCase):
    """
    A Prof Of Concept for creating one generalized GPXReader test class.
    It will then be inherited by other classes that will use TrackPoints example data,
    Route Points data and Way Points data using pytest-fixtures.
    """

    @parameterized.expand([
        ('web/tests/example_data/fast_example.gpx'),
        ('web/tests/example_data/fast_example_trackpoints.gpx'),
        ('web/tests/example_data/fast_example_waypoints.gpx'),
        ('web/tests/example_data/fast_example_routepoints.gpx'),
    ])
    def test_get_points(self, gpxfile):
        """
        Tests if there will be data output from get_points.

        Test if:
        - points exists for different types.
        - what will happen if types exists, but no data?
        """
        gpxreader = GPXReader(gpxfile)
        points = []
        for point in gpxreader.get_points():
            points.append(point)
        assert len(points) > 0

    @parameterized.expand([
        ('web/tests/example_data/fast_example.gpx'),
        ('web/tests/example_data/fast_example_trackpoints.gpx'),
        ('web/tests/example_data/fast_example_waypoints.gpx'),
        ('web/tests/example_data/fast_example_routepoints.gpx'),
    ])
    def test_get_elevations(self, gpxfile):
        """
        Tests if all elevations will be returned from input element.
        Test should check if elements are TrackPoints Route Points or Way Points.
        """
        gpxreader = GPXReader(gpxfile)
        elevations = gpxreader.get_elevations()
        assert len(elevations) > 0

    @parameterized.expand([
        ('web/tests/example_data/fast_example.gpx', 224),
        ('web/tests/example_data/fast_example_trackpoints.gpx', 224),
        ('web/tests/example_data/fast_example_waypoints.gpx', 224),
        ('web/tests/example_data/fast_example_routepoints.gpx', 224),
    ])
    def test_get_elevations_len(self, gpxfile, elevations_len=224):
        """
        Test if proper amount of elevations are read by get_elevations method.
        """
        gpxreader = GPXReader(gpxfile)
        elevations = gpxreader.get_elevations()
        assert len(elevations) == elevations_len

What I don't like about this source-code?

I don't like that there is a lot of repeated code (if parameters are a repeated code ? ) that leads to break the DRY Principle

I'll need to think if this is actually a "repeated" code or if some of this functions does not need so much of parameter types.

Code commits done for this post:

Tools and applications used:

VIM + Plugins (Fugitive, Vim-Markdown and Vimux )
Docker
Tmux

Accomplished:

1. Add test-cases for business logic to GPXReader - endorphines-algorithms :)

What's next

1. Python backend - initial REST-API.

2. Making API documentation - research for best API documentation!

GPXReader - Clearing TDD Tests

2017-04-15T18:00:00+02:00

Introduction

As you might probably read at yesterdays article I've finally found how I shoud follow TDD principle and "flow".

It basically means, I will not create a bunch of tests, but instead I'll create a bunch of not-implemented methods/functions that are documented about their responsibilities but has no code inside of them.

After that I'll start working on those methods with a TDD.

Prerequisites - Clearing up pre-existing tests.

1. Why your unittests should be fast?

I'll make an example of how my testing failed in real-life example. I was working on a python project. In that project I didn't follow the TDD rules, but instead, once the "feature" has been created I created tests that "covered" expected behaviour of source-code.

These tests were not time-optimized, they used real-time data and needed access to internet service.

That made them a lot time consuming. It also pushed me to not run all of the "unit-tests" but only those that I was working currently, and then before "pushing" work of the tests, check if other tests are working properly.

Now as I recall it was not such a bad thing, since I didn't use TDD properly and source-code of other features was not changed in some new "features".

But in some occasional cases I needed to change behaviour of features that had already test-coverage, and then I needed to run all tests that took a bit of time.

Then, instead of working on tests I was waiting actually until tests will complete with success or failure. To make a more expressive example, please check following:

Each run of all tests took around 120 seconds (2 minutes) (yeah .... SECONDS... :( ) - in earlier phase of project, at the end of project it was close to 300 seconds (5 minutes... )
Each feature per day might took at least 5 runs of tests-checking to find if any regression is not on source-code.

This takes 120 * 5 = 600 seconds = 10 minutes of (waste? ) time (in early phase of project)

In the end it took: 300 * 5 = 1500 seconds = 25 minutes of overall interrupted idle-waiting periods.

Of course I took this time to re-read source-code, but either way I was in a "waiting" stage.

It's an optimistic prognosis. What If your 8h work day makes about 20-25 changes and each change you need to run all tests ?

Easy to measure : 25 * 300 = 125 minutes! = more then 2 hours of (wasted?) time.

2. What I've found out in code?

In my GPXReader Tests I've found a bunch of tests that were not implemented. Those were just information for myself to speed-up development with TDD and pomodoro-technique.

I've figure-out that this information is more-less self-explanatory and It's more a documentation for source-code than test-documetation.

Check below how it look like before changes :

Link to full source-code at github

class BaseGPXReaderTest(object):
    """
    A Prof Of Concept for creating one generalized GPXReader test class.
    It will then be inherited by other classes that will use TrackPoints example data,
    Route Points data and Way Points data using pytest-fixtures.
    """

    def test_get_points(self):
        """
        Tests if there will be data output from get_points.

        Test if:
        - points exists for different types.
        - what will happen if types exists, but no data?
        """
        raise NotImplementedError()

    def test_get_elevations(self):
        """
        Tests if all elevations will be returned from input element.
        Test should check if elements are TrackPoints Route Points or Way Points.
        """
        raise NotImplementedError()

    def test_get_elevations_len(self, elevations_len):
        """
        Test if proper amount of elevations are read by get_elevations method.
        """
        raise NotImplementedError()

    def test_get_track_points(self):
        """
        Test if all track points are gathered
        """
        raise NotImplementedError()

    def test_get_route_points(self):
        """
        Test if all route points are gathered
        """
        raise NotImplementedError()

    def test_get_way_points(self):
        """
        Test if all way points are gathered
        """
        raise NotImplementedError()

    def test_get_lowest_elevation(self):
        """
        Checks method for:
        - If no elevations in route, method should return None.
        - If elevations are equal or only one available, return None.
        - If more then one elevation (different) available, return lowest.
        """
        raise NotImplementedError()

    def test_get_highest_elevation(self):
        """
        Checks method for:
        - If no elevations in route, method should return None.
        - If elevations are equal or only one available, return None.
        - If more then one elevation (different) available, return highest.
        """
        raise NotImplementedError()

    def test_animal_figure_route(self):
        """
        Tests if method named "animal_figure_route" will return proper name of animal
        for proper biking route.
        """
        raise NotImplementedError()

class AbstractGPXReaderTest(BaseGPXReaderTest):
    """
    Abstract Implementation of Base GPX Reader Test
    """

    def test_get_points(self):
        """
        Tests if there will be data output from get_points.

        Test if:
        - points exists for different types.
        - what will happen if types exists, but no data?
        """
        points = []
        for point in gpxreader().get_points():
            points.append(point)
        assert len(points) > 0

    def test_get_elevations(self):
        """
        Tests if all elevations will be returned from input element.
        Test should check if elements are TrackPoints Route Points or Way Points.
        """
        elevations = gpxreader().get_elevations()
        print len(elevations)
        assert len(elevations) > 0

    def test_get_elevations_len(self, elevations_len):
        """
        Test if proper amount of elevations are read by get_elevations method.
        """
        elevations = gpxreader().get_elevations()
        assert len(elevations) == elevations_len

    def test_animal_figure_route(self):
        """
        Test implementation for animal figure route
        """
        raise NotImplementedError()

@pytest.mark.usefixtures("gpxreader")
class TestTrackPointGPXReader(AbstractGPXReaderTest):
    """
    Specific version of Abstract test-class that uses mainly TrackPoints gpx file for tests.
    """
    pass

3. What changes I've made to make this more clear?

So I've change this source-code and ripped it apart.

I've changed the BaseGPXReaderTest with functions into a Abstract class for GPXReader with functions that needs to be implemented:

Link to full source-code of this file at github

class AbstractGPXReader(object):
    """
    AbstractGPXReader - An Abstract Class containing most of methods that needs to be implemented.

    I.e.
    get_track_points(self, item_nb=0):
    """

    def get_track_points(self, item_nb=0):
        """
        Track points are gathered
        """
        raise NotImplementedError()

    def get_route_points(self, item_nb=0):
        """
        Route points are gathered
        """
        raise NotImplementedError()

    def get_way_points(self, item_nb=0):
        """
        Way points are gathered
        """
        raise NotImplementedError()

    def get_lowest_elevation(self):
        """
        Returns information about lowest elevation from route
        Returns None if not found any elevations!

        Checks method for:
        - If no elevations in route, method should return None.
        - If elevations are equal or only one available, return None.
        - If more then one elevation (different) available, return lowest.
        """
        raise NotImplementedError()

    def get_highest_elevation(self):
        """
        Checks method for:
        - If no elevations in route, method should return None.
        - If elevations are equal or only one available, return None.
        - If more then one elevation (different) available, return highest.
        """
        raise NotImplementedError()

    def animal_figure_route(self):
        """
        Uses Points to compare route with animal shapes.
        Returns information if this route looks similar to some animal or not.
        Returns Name of animal from enumeration or None if none animal found.

        Tests if method named "animal_figure_route" will return proper name of animal
        for proper biking route.
        """
        return NotImplementedError()

Final result

For now final result does not look "much" better. Instead it's a fast implementation! :)

It follows TDD technique in at least point of "fast-unit-tests".

To make unit-tests faster for now, I"ve changed the .gpx example file smaller and faster to run.

I've also find that the Makefile is not best optimized in terms of timing.

Makefile contained pip install -r requirements additionally at the "make unittest" flow , that is not needed at all if you already have pip-installed all dependency.

So changes I've made in Makefile:

unittest_all: prepare_db unittest

unittest:
    cd bikingendorphines && python manage.py test

Now all I have to do is follow TDD technique and make tests for our GPXReader.

Code commits done for this post:

Tools and applications used:

VIM + Plugins (Fugitive, Vim-Markdown and Vimux )
Docker
Tmux

Acknowledgements

Tomatoid - a Pomodoro-Technique web-based(and not only) web-service.

Accomplished:

1. Add even more tests to GPXReader.

Created additional markdown_commits script that transforms git-log into Markdown links :) check HERE

What's next

2. Add test-cases for business logic to GPXReader - endorphines-algorithms :)

3. Python backend - initial REST-API.

4. Making API documentation - research for best API documentation!

FancyBox Plugin and Test Driven Development Flow

2017-04-12T15:30:00+02:00

Test Driven Development Flow

So lastly I've introduced to myself a Test Driven Development, but unfortunatelly I didn't examine the flow and approach as much as I should.

Thanks to Uncle Bob I've realized what exactly TDD flow looks like.

1. Make a test that (should) fail.

The idea is not to make any type of source code before test-code, but instead, create a test that probably will fail if you haven't got any source-code.

2. Pass your failing test

Make your initial test not failing by creating just enought source-code for this particular test.

3. Refactor your source code.

Then finally refactor your code to look better.

4. Repeat steps 1-3 untill your source-code does what you expect it from.

Check this awesome image that will help me with TDD approach in this initial days of working in this system:

TDD Global Lifecycle by Wikipedia

What I did wrong this time ? :)

Well... this time I've figure out that I'll make a bunch of tests for all bunch of functions that will help me focus on small elements in a short period of time (pomodoro technique-remember?)

Then I ofcourse would normally start working on this tests to make a TDD approach.

But let's follow TDD in a different way.

Let's use this information I've gathered as more "documentation" of what should be done, then how this code should look like.

Initial test.

Lets first create a test for our plugin:

def test_plugin_registers():
    """
    Check if plugin registers to article_generator_finalized
    """
    register()
    assert_receiver_registered(fancybox_plugin)

And then make source-code that will pass test (that's a refactored code):

def register():
    "Registers plugin"
    signals.article_generator_finalized.connect(fancybox_plugin)

def assert_receiver_registered(receiver_name):
    'Checks if receiver is registered to signals generator'
    assert signals.article_generator_finalized.has_receivers_for(receiver_name)

def test_plugin_registers():
    """
    Check if plugin registers to article_generator_finalized
    """
    register()
    assert_receiver_registered(fancybox_plugin)

Final result

I've made thus far 3 tests that checks few simple but needed things.

For now final result looks like this:

"""
FancyBox unittests
"""
from pelican import signals

class Article(object):
    "A simple Article class"
    content = ""

def article_generator():
    "Article generator"
    yield Article()

def fancybox_plugin():
    "Fanxybox plugin - temporary code placement"
    pass

def register():
    "Registers plugin"
    signals.article_generator_finalized.connect(fancybox_plugin)

def assert_receiver_registered(receiver_name):
    'Checks if receiver is registered to signals generator'
    assert signals.article_generator_finalized.has_receivers_for(receiver_name)

def test_plugin_registers():
    """
    Check if plugin registers to article_generator_finalized
    """
    register()
    assert_receiver_registered(fancybox_plugin)

def test_article_generator_return_article():
    """
    Checks if generator return article
    """
    assert isinstance(article_generator().next(), Article)

def test_given_article_generator_check_article_content_exists():
    "Checks if article content field exists in article"
    for article in article_generator():
        assert hasattr(article, 'content')

Code commits done for this post:

Tools and applications used:

vi/vim
pytest
docker
tmux

Accomplished:

1. Make tests for fancybox plugin

What's next

1. Make more tests and source-code of fancybox plugin

2. Start working on plugin for asciinema player.

3. See ISSUES task-list for this 'mini' growing project :)

GPXReader - Business Logic and Tests

2017-04-09T20:00:00+02:00

Introduction

I've informed you I'll be doing Business Logic for Biking Endorphines.

To make best of my time I've considered to go with TDD and that's why I'll first implement Tests for Business Logic.

This will propably shape all business logic, but I'll have already tests with myself for future.

Prerequisites - Business Logic

1. Business logic for Biking-Endorphines.

So the Business Logic for Biking Endorphines is going to be Endorphines badges algorithms themselves.

Just as I've revealed in behind-closed-doors, type of algorithms for endorphines badges will be:

Adventure style - Calculate how you could change your route to (shorten/lenghen/alternate) your ride-path
Training style - to keep up with every-day motivation for training - i.e. showing how your timing can change by training today.
Family style - to find exciting places that you didn't know about and to share this experience with family.

And others that I currently haven't figure-out yet.

2. Why I will start my implementation of B-Logic with tests?

So first of all, the TDD approach in my opinion is something beautiful in it's core. You first create tests for functions/methods/classes that will make your b-logic.

This tests will fail at beginning. But they will also keep you motivated to keep-up with code and give you path by which you should follow when creating code-itself.

Lately I've watched YT talk about Super is Super - by Raymond Hettinger That I was really impressed about!

I might think about using TDD and the Super is Super in my implementation.

Final result

My limited resources made me create only simplest tests for pre-existing code and some "NotImplemented" tests that will need to be created in future.

I've found out that if you put "NotImplementedError" as return in tests, this test-function than does not fail!

Code commits done for this post:

Adds few new tests for gpxreader

Adds elevation tests for gpxreader

Tools and applications used:

VIM + Plugins (Fugitive, Vim-Markdown and Vimux )
Docker
Tmux

Accomplished:

1. Add more tests to GPXReader test cases.

What's next

1. Add even more tests to GPXReader.

2. Add test-cases for business logic to GPXReader - endorphines-algorithms :)

3. Python backend - initial REST-API.

4. Making API documentation - research for best API documentation!

Biking Endorphines - Introduction to Android sub-project

2017-04-06T00:00:00+02:00

Introduction - What's this Android application doing in python project??

You may wonder why use Android application in python project??

Python project was from the beginning only one part of product that I wanted to create.

Second part was always the Android application that will gather data from GPS and show information about Endorphines Badges.

Prerequisites - Android-Studio

1. How to install Android Studio (on Ubuntu)

Please download the Android Studio 2.3 (~429MB)
Unpack files with unzip android-studio-ide-162.3871768-linux.zip
Install JRE if you have not yet done it by sudo apt-get install -y default-jre && sudo apt-get install -y default-jdk
- If for some reason your default-jdk is below Android-Studio requirements, try to use this:

sudo add-apt-repository ppa:webupd8team/java -y
sudo apt-get update
sudo apt-get install oracle-java8-installer

sudo apt-get install oracle-java8-set-default

Check version:

java -version

If you have already installed Java Runtime Environment (JRE) then after typing "sh android-studio/bin/studio.sh" you should see IDE start running.
If after installing JRE you see error that says No JDK found. Please validate either STUDIO_JDK, JDK_HOME or JAVA_HOME environment variable points to valid JDK installation, don't panic.
- Try changing JDK_HOME system variable to proper path of jdk i.e. JDK_HOME=/usr/java/jdk1.7.0.21/

2. Android introduction - how to start your first project

Take few minutes of your time and explore "Create an Android Projects" - This will give you the most basic information about how to startup first project with Android Studio.

3. What should be in your git-repository?

All your files that are project resources.

GitIgnore can be generated with gitignore.io i.e. for android, android-studio and intellij.

If you accidentally pushed build files, first do :

Gradle build:clean task - this will remove this files so you know what files are not-needed in git status
use bfg repo cleaner as described here
commit and push cleaned commits to your remote repository

4. Why not use vi/vim and terminal for all your android-development ?

Well... you can certainly use vim for development, but vim-itself is not perfect IDE for the job. You should at least get yourself around some vim-plugins that could simulate some of the AndroidStudio behaviour, i.e. Gradle usage:

Vim-Android Plugin by hsanson

Final result

1. What I've created as initial?

Separated git repository available here.
Adding android application repository to main repository as a module
- If you want to know how to do that, please follow this link - for those impatient, use git submodule add GIT-LINK [optional directory to which clone repo].
- i.e. git submodule add https://github.com/anselmos/Biking-Endorphines-Android-App androidapp
Initialized clean empty project for androidapp with minimum API 16(Android 4.1 - JellyBean) - since I've got one device that I'll use for tests and according to Google Play Store information about 95% of devices uses this minimum version of API.
Setup en Empty Activity for startup.

2. What are the plans for this android-sub-project?

Layout :
- information about gps -is it on, active and properly initialized
- starting and stopping/pausing button for biking activity
- statistics mocked for this time, but eventually gathered from rest-api.
Start/Stop gathering data from gps as a GPX file format
Make Login-page that will login android-device in backend as a device-id, timestamp, user-id
Gather all information about statistics with endorphines-badges!
Presentation of Endorphines Badges with fun icons.
Divination - Tea-Leaf Reading for your biking-endorphines-badges - what you could do to gather more badges ! - based on analysis of bike-rides. - a pseudo AI for badges gathering

Code commits done for this post:

Tools and applications used:

Android Studio
VIM + Plugins (Fugitive, Vim-Markdown and Vimux )
Docker
Tmux

LL (Lessons Learned)

1. I've added git submodule in a bad way.

Initially I've added android application to project repository in a bad way - I've forgot to add directory path to which submodule should be cloned.

So I've decided to run rm -rf on git-submodule files and the gitsubmodule config file.

Unfortunatelly when I tried to use git add submodule again, I've crossed at error like this :

A git directory for '[HERE GOES NAME OF DIRECTORY]' is found locally with remote(s):

The problem was almost identical as described at stackoverflow

Solution:

I needed to remove .git/submodule/[Directory-containing-submodule] to make git forget about my first attempt to clone submodule.

Acknowledgements

Accomplished:

1. Android initials - how to start-up with Android Projects - and that sort of things.

2. Android Sub-project - plans and draft of application.

What's next

1. Add more tests to GPXReader test cases.

2. Add more business logic to GPXReader - endorphines-algorithms :)

3. Python backend - initial REST-API.

4. Making API documentation - research for best API documentation!

Pelican Blog with Fancy Box image viewer

2017-04-05T00:00:00+02:00

Introduction

As I've informed you at post Pelican Blog with Pelican-Bootstrap3 theme, I've started working on fancy-box image viewer and adjusting it's usage for pelican blogging engine.

FancyBox - What it is and how it works

Quoting product site:

FancyBox is a tool for displaying images, html content and multi-media in a Mac-style "lightbox" that floats overtop of web page. It was built using the jQuery library. Licensed under both MIT and GPL licenses

Product page here

FancyBox - How to use it?

FancyBox uses jQuery as main dependency. For using Fancybox you have to add it to your head/body of web-page.

Here is a simple code-block you can use:

<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4/jquery.min.js"></script>

Also you have to add Javascript Fancybox script (CDN recommended to download before):

<script type="text/javascript" src="/fancybox/jquery.fancybox-1.3.4.pack.js"></script>

CDN for fancybox you can find here: https://cdnjs.com/libraries/fancybox

And finally the CSS style file:

<link rel="stylesheet" href="/fancybox/jquery.fancybox-1.3.4.css" type="text/css" media="screen" />

What are pelican-plugins and how to use it?

Pelican Plugins are just a regular python files, but added in pelicanconf file as a "PLUGINS". They extend functionality of original Pelican engine by using the signals - so registering on a signal type - for more information about that please check pelican plugin site

FancyBox - why create a plugin out-of-it ??

First of all - because I haven't done any plugins at all yet - sounds like fun and a lot of try-and-error :)

Besides - I don't want to have my script incorporated in Markdown file each time I want to add image to it.

Plugin will make that work for me ! (laziness is for virtues programmers )

Fancybox plugin - I've created a separated github-project for that!

I've decided to move FancyBox plugin to separated github-project for convenience - in that situation no dependency between version of pelican and plugin will exist.

Fancybox initial approach

As usually I've started coding without re-thinking. And taking into account my time-limits it was a bit of failure.

I've almost found solution and it almost worked, but it was a lot of copy-paste that I don't really like and apprieciate.

So I've figure-out to change my approach after I've seen polish programmer/blogger Maciej Anisewicz's vlog at Youtube about "programming flow is a bullsh*t".

It's a polish programming talk that to summarize - says about TDD as better approach for programming then having all information about program-flow in your mind, and about making notes for all your ideas.

Fancybox final approach

I've decided to go additional way in my programming approach - to make best out of the TDD.

That's why I'll first create tests for code and then create code itself. It'll be a longer way, but in that solution I'll use pomodoro-technique much more efficient.

I've created a stubs for my tests that have a "NotImplementedError" raised. Once I'll finish with tests I'll then start coding.

Let's finally find-out if the so called "TDD" is worth using or not :)

Code commits done for this post:

A Github project for plugin: Pelican-Fancybox

Commits:

Pelican-Blog Pull Requests

Made pull-request for theme - #5

Tips & Tricks

1. TDD approach

Make tests that will fail
Make code and fix code so that tests will pass
Commit your fully TDD certified code!! :)

Tools and applications used:

Vi/vIM
Docker

Accomplished:

1. I'll continue working on pelican-theme with using `fancybox`

Acknowledgements

What's next

1. Make tests for fancybox plugin

2. Start working on plugin for asciinema player.

3. See ISSUES task-list for this 'mini' growing project :)

Pytests Unit Testcase for Biking Endorphines

2017-03-31T00:00:00+02:00

Introduction

So my first initial change for Parameterizing test-cases was a bit of failure - you can check this Parameterized Django Testcases At Biking Endorphines

Let's change that :)

Prerequisites - How to use Pytests

1. Install pytests with pip :

pip install pytest

2. Import Pytest in your file containing tests:

import pytest

3. But how do you actually use it ?

That's a good question.

First create a file with different name then pytest.py i.e. first_test.py and put similar content as I've created :

"""
Pytests example
"""
import pytest


def func(num):
    """
    Simple adding function for pytest showcase
    """
    return num + 1

def test_answer_fail():
    """
    Test for checking func with negative values
    """
    with pytest.raises(AssertionError):
        assert func(3) == 5

def test_answer_success():
    """
    Test for checking func with possitive values
    """
    assert func(4) == 5

Then use following bash command:

pytest -rs first_test.py

That should make following output:

pytest -rs first_pytest.py 
============================================================================================================ test session starts ============================================================================================================
platform linux2 -- Python 2.7.6, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /home/bawi/projects/deployed/anselmos_blog/data/content/examples/pytest, inifile:
collected 2 items 

first_pytest.py ..

========================================================================================================= 2 passed in 0.02 seconds ==========================================================================================================

What changes I've made in code - a diff between code-commits using asciinema and vim-diff tool?

1. I've followed Good Integration Practices

What that means ? I've figure-out that this tests that I intent to create are out-side of django scope, so why not just put them literally out-side?

That's why I've created tests directory at directory bikingendorphines/tests and put all my test-files in there.

2. Test-Case itself.

Now I've change code so it looks like this :

"""
Tests for GPXReader class
"""

import pytest

from web.utils import GPXReader

@pytest.fixture(
    scope='module',
    params=[
        'bikingendorphines/example_data/15212277.gpx',
        'bikingendorphines/example_data/15212277.gpx'
    ]
)
def gpxreader(request):
    """
    Creates GPXReader instance with predefined gpx file
    """
    return GPXReader(request.param)

# pylint: disable=redefined-outer-name
def test_get_points(gpxreader):
    """
    Tests if there will be data output from get_points.

    Test if:
    - points exists for different types.
    - what will happen if types exists, but no data?
    """
    points = []
    for point in gpxreader.get_points():
        points.append(point)
    assert len(points) > 0

Final result

Check how much my code changed in this commit that shows all diffs:

Adds pytest. Adds GPXReader pytest-alike test

Code commits done for this post:

That's the same commit that changes GPXReader:

Adds pytest. Adds GPXReader pytest-alike test

And I've finally Squashed and Merged the gpx-solution as a initial solution.

Now I hope Travis will make my day better and confirm pylint&unittest as a good-to-go and create Pyreverse GH PAGE :)

Check results of Travis here

Update : Yes :) Travis accepted my changes and created pyreverse output in gh-pages, see GH_PAGE, Classes as PNG File

Tools and applications used:

VIM + Plugins (Fugitive, Vim-Markdown and Vimux )
Docker
Tmux

LL (Lessons Learned)

1. Don't try to name your file with test-cases a `pytest.py` ! :)

If your filename will have name pytest.py as I did for my example of usage pytest, I've found python mismatched pytest and was confused.

Error like this popup:

________________________________________________________________________________________________________ ERROR collecting pytest.py _________________________________________________________________________________________________________
import file mismatch:
imported module 'pytest' has this __file__ attribute:
    /home/bawi/projects/deployed/anselmos_blog/data/content/examples/pytest/.env/local/lib/python2.7/site-packages/pytest.py
which is not the same as the test file we want to collect:
    /home/bawi/projects/deployed/anselmos_blog/data/content/examples/pytest/pytest.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

Solution:

You may overcome this issue change your name to something different than name of the library itself :)

2. Create plan and stick to it

That's more a Tips&Tricks then Lessons Learned, but it may also count as LL. I've figure-out finally how to manage the planning - I'll try my method and let you know if it will actually work:

Create a .md file for article
Make only main parts information of article that will briefly describe a kind-of-todo
Make all of this parts and describe them more deeply.

Acknowledgements

Accomplished:

1. Change behaviour of GPXReaderTestCase to more pytest alike and add some more test code

2. Push finally first version of GPX Reader to master via pull-request.

What's next

1. Add more tests to GPXReader test cases.

2. Add more business logic to GPXReader - endorphines-algorithms :)

Parameterized Django Testcases At Biking Endorphines

2017-03-30T00:00:00+02:00

Introduction

After reviewing what is at the gpx_reader branch I've found #TODO that said about making mocks, but it was not best name for what I've intended to do. I've wanted to parameterized the TestCase, so I could use different type of test-source data for one and the same test source-code.

In my professional work I've already done something similar, so figuring this out was not hard, hence I've found the parameterized library and used it.

Prerequisites - How to use Parameterized

Parameterized is a python library that usage is pretty simple.

1. Install Parameterized with pip:

pip install parameterized

2. Import Parameterized in your tests*.py files:

from parameterized import parameterized

3. Usage - here we go!

So how to use it?

Well taking into account I'm working with django.TestCase that is a deriviation of unittest.TestCase I found information in Parameterized of usage that said:

import unittest
from parameterized import parameterized

class AddTestCase(unittest.TestCase):
    @parameterized.expand([
        ("2 and 3", 2, 3, 5),
        ("3 and 5", 2, 3, 5),
    ])
    def test_add(self, _, a, b, expected):
        assert_equal(a + b, expected)

For other usages you may find Parameterized Projects in Github as a documentation.

What I've done with code:

In my case I've wanted to include parameters for gpx_file at setUp, but it was to hard and time-consuming to find out how to make it work.

A semi solution I've found that is not pretty, but works is as presented below:

class GPXReaderTestCase(TestCase):
    """
    Tests all GPXReader class cases for methods access.
    """
    @parameterized.expand([
        ('bikingendorphines/example_data/15212277.gpx'),
    ])
    def test_get_points(self, gpx_file):
        """
        Tests if there will be data output from get_points.

        Test if:
        - points exists for different types.
        - what will happen if types exists, but no data?
        """
        self.gpxreader = GPXReader(gpx_file)
        points = []
        for point in self.gpxreader.get_points():
            points.append(point)
        self.assertGreater(len(points), 0)

Why this code is not pretty??

Well, you find yourself in situation, when for each test you will have to add information about GPXReader. Using setUp with this field would make creating code a little bit easier to maintain and read.

I'll probably go back to this code and use the cleaner pytest base which may clear this small issue.

Code commits done for this post:

Adds mocking of gpx file using parameterized.

Tools and applications used:

VIM + Plugins (Fugitive, Vim-Markdown and Vimux )
Pelican
Docker
Tmux

LL (Lessons Learned)

1. Pick tools for what they are intented for.

I've forgot that default django.TestCase are for testing Django-framework itself behaviours like ORM inside of django and other.

They are not ideal for unit-testing and for that you should use something different.

Solution:

For unit-testing better to use one of those :

Acknowledgements

What's next

1. Change behaviour of GPXReaderTestCase to more pytest alike and add some more test code

2. Push finally first version of GPX Reader to master via pull-request.

3. Check Issues of this project.

Pelican Blog with Pelican-Bootstrap3 theme

2017-03-29T21:30:00+02:00

Introduction

As I've informed you at post Pelican blog with Fabric and FTP deployment part 2/2 I've started working on Pelican Bootstrap3 Theme.

it's a MIT licensed theme that supports categories, github project's information, social information and few others that I'm currently not interested in.

How to use pelican-bootstrap3 theme

To use this theme, first thing you need is to git clone this repository:

git clone https://github.com/getpelican/pelican-themes/ themes

Then add information about path for your selected theme in pelicanconf.py as presented below:

PLUGIN_PATHS = ['plugins']
PLUGINS = ['i18n_subsites']
THEME = "themes/pelican-bootstrap3"
JINJA_ENVIRONMENT = {
    "extensions": ['jinja2.ext.i18n'],
}

Then please install i18n plugin from here, using below mini-script (unfortunatelly they all are in one repository, not as git sub-modules.)

git clone https://github.com/getpelican/pelican-plugins plugins

You may want to remove all other plugins besides i18n_subsites:

cd plugins && mv i18n_subsites ../ && rm -rf . && mv ../i18n_subsites .

And it will start working.

What tips and tricks I've checked with this theme:

1. Github Repositories.

Great idea that you can use your github account and just use without additional plugins gh-repos visible at right banner.

You may want to trick it to only view your own source projects with:

GITHUB_USER = 'anselmos'
GITHUB_SKIP_FORK = True

2. About me banner information.

There is an information for something similar to About-Page, but available all-the-time in right-banner, it's "ABOUT_ME".

Just add in your pelicanconf.py variable ABOUT_ME that will contain all information you want to say about yourself. This variable will also properly decode all html-type metadata.

3. Your code-blocks are customizable!!!

You can change your code-blocks style to different if you want. There are a lot of them because it's using Pygments.

To check what type of Pygments style you can use, check pelican-bootstrap3-theme-page

4. Many many other options

This options can be setup, by reading first information about theme that is available here.

Code commits done for this post:

Tools and applications used:

Vi/vIM
pelican
Docker

What's next

1. I'll continue working on pelican-theme with using `fancybox`

2. See ISSUES task-list for this 'mini' growing project :)

GPX Initial Solution

2017-03-26T22:30:00+02:00

Plan for this solution

This time I had a plan.

The plan assumed that I'll create initial solution for GPX reading information.

Reading basic information like points and just create initial test-cases for this.

I've even created a sub-class of GPXReader for Endomondo Specific format data, but thus far endomondo's format data is almost identical with my example data.

Initial Solution - ideas behind the curtain

Solution is simple - reading points from what-ever data is in gpx file i.e. routepoints, trackpoints, waypoints.

Solution is far from perfect, but this time I made it in my timeframe I've given to myself - YEAH!!

Check this source code which is the core for gathering points: (for today - it may vary at the end)

    def get_track_points(self, item_nb=0):
        """
        Returns only first track points
        """
        points = []
        if len(self.__tracks) > 0 and len(self.__tracks) >= item_nb:
            for point in self.__tracks[item_nb].walk():
                points.append(point)
        return points

    def get_route_points(self, item_nb=0):
        """
        Returns route points
        """
        points = []
        if len(self.__routes) > 0 and len(self.__routes) >= item_nb:
            for point in self.__routes:
                points.append(point)
        return points

    def get_way_points(self, item_nb=0):
        """
        Returns way points
        """
        points = []
        if len(self.__waypoints) > 0 and len(self.__waypoints) >= item_nb:
            for point in self.__waypoints:
                points.append(point)
        return points

    def get_points(self, item_nb=0):
        """
        Returns Points from either route/track or waypoints.
        If there is more then one type then uses Track's as a default.

        Make assumption that there will be only one of each data_type
        i.e. only one track/route/waypoint.
        """
        if len(self.__tracks) > 0 and self.__tracks[item_nb].get_points_no() > 0:
            return self.get_track_points()
        elif len(self.__routes) > 0 and self.__routes[item_nb].get_points_no() > 0:
            return self.get_route_points()
        elif len(self.__waypoints) > 0 and self.__waypoints[item_nb].get_points_no() > 0:
            return self.get_way_points()
        else:
            return []

Code commits done for this post:

Commits in PR for GPX Reader, and it will not yet be pulled.

Adds initial Utils.pY

Adds more information on gpx reader at utils

Initial GPXReader with Test-Cases

Adds fix for unused import of gpx

Tools and applications used:

vIM
pylint/django tests
make
Docker
Github pull-request

LL (Lessons Learned)

1. I've created test-cases before checking if code even works :)

You may find yourself creating tests for code, before you even run it manually, as I did.

Then unfortunatelly debugging inside of test-cases is not ideal.

For now with limited time, I didn't find solution that could use django tests with code and debugging code within tests.

Solution:

First test if your solution works - manually, then create test-case. When creating test-case source-code may or may not be changed, but you can check it with your manual testbed.

Acknowledgements

GPS Waypoints, Routes, and Tracks – the difference

Pull Request I've created today for GPX Reader

What's next

1. I'll continue working on GPX Reader to improve code for better maintainance and test-coverage.

2. Probably move again to travis builds to make pyreverse builds per commit visible at github gh-pages #12.

3. As always - check my Issues for the project

GPX File Reader

2017-03-26T10:30:00+02:00

GPX File Reader - initial requirements.

Yeah so as I've announced at Github Pages and Travis auto Deployment, I'm starting work on GPX File reader.

Let's first create a requirements on this task.

What's our assumptions ?

1. We will not invent wheel again

I'm not going to create another library that handles reading gpx files them-selves, since It's not the main activity for the project.

I'll focus on reading information that maybe of great interest for this particular project.

2. GPX Reader will support different gpx-formats

Reader will need to support different type of gpx-file formats. Starting from Endomondo/Strava/ and alternatives-to-endomondo stopping at (if needed) my own gpx-format created by Android-sub project.

I do hope, that library it-self will support it, but taking into account that gpx files are just an xml files, I have high-hopes.

3. GPX Reader needs to be extensible via plugins-engine

Project assumes that this reader will not be "universal", and being so- I'll maybe need to use it someday (even in different type of project) and then I'll just create a plugin to already working solution.

Besides I haven't yet done something like plugin-engine extensions - and it looks like something very interesting.

4. What data this project needs ?

For this project I'll read information like gpx locations, timings between each location, routes as a bunch of locations. That data will be very important and can be used for statistics analysis and - story-telling - for gathering endorphine-badges.

What are the engines to read gpx files?

1. Manual reading information from xml

GPX files in it's core are just xml files, hence you can read them as usual xml files.

Check this libraries that handles xml read:

XML-eltree
LXML
BeautifulSoup
xmltodict
and many other similar xml-reading libraries.

But BE-AWARE! - this maybe harmfull to your neuronic-system and cause some neurotic issues :) since you'll need to scrap all valid information i.e. like all coordinates for route by yourself manually with just xml library.

2. GPXPY library

Some of the features ( i.e. gathering routes not by using XPATH search )that this particular library contains, makes it almost perfect match for this particular job.

GPXPY

So I'll stick to it. As I do recall properly I've been using this library for tests in my previous 'abandoned' project - the test file.

Prerequisites - How to use GPXPY ?

1. Installation of gpxpy

First of all you need to install python library:

pip install gpxpy

Then JUST USE IT .

Oups sorry, I've meant this link - with example

as a Filipe Fernandes uses it.

2. Find gpx file examples to work with - or train with endomondo and create your-own.

I've searched web and finally I've found nice site that gives some gpx files with routes other users went that you also could upload to your Endomondo/Strava application and "go with this route".

Using this GPX file

3. Example code to check functionality of gpxpy.

import gpxpy
import gpxpy.gpx
gpx_file = open('15212277.gpx', 'r')


# Parsing an existing file:
# -------------------------

gpx = gpxpy.parse(gpx_file)

for track in gpx.tracks:
    for segment in track.segments:
        for point in segment.points:
            print 'Point at ({0},{1}) -> {2}'.format(point.latitude, point.longitude, point.elevation)

for waypoint in gpx.waypoints:
    print 'waypoint {0} -> ({1},{2})'.format(waypoint.name, waypoint.latitude, waypoint.longitude)

for route in gpx.routes:
    print 'Route:'
    for point in route.points:
        print 'Point at ({0},{1}) -> {2}'.format(point.latitude, point.longitude, point.elevation)

# gpx_parsed = gpxpy.parse(open(gpx_file)) # This will fail with error of coercing which means that you already have file opened for reading and trying to open it again, those failing.

Source-Code.Py

GPX-example-file

4. Where to put your business logic code for importing data from gpx files at Django Framework?

That was the hardest question I could not find clear answer - Then I've finally found out why - there is no good "answer for all business logics"

It all depends on what your business logic will do. If it has any type of connections with Model and will reuse of model-api than you should put your business logic there. The same applies to Views.

The only Rule of Thumb you should follow is not to overload model/view or controller.

Since our importing methods will only read data from gpx files, I've decided to move to utils.py file as suggested here -> business-logic-in-django-at-stackoverflow.

Geo-Location and other possibilities out-there.

While investigating python gpx libraries I've found library that I might use for GeoCoding.

It's a GeoPy - a Geocoding Library for Python - quoting authors.

I've also found that there is an Geo-framework for Django called GeoDjango

Code commits done for this post:

Unfortunatelly None. I didn't have proper time to sit and code with gpxpy. That's why I introduce "Things I've found worth looking into" :)

Tools and applications used:

Vi/Vim
VimFX plugin for firefox
Docker

Things I've found worth looking into.

1. PyTrainer

Wow I've just found some interesting project called PyTrainer - quoting official github repo info: Pytrainer is a tool to log all your sport excursion coming from GPS devices (with a focus on ForeRunner) or GPX (http://www.topografix.com) files. Pytrainer supports GPS track files and displays it in graphs, maps...

So that's definatelly something to look deeply into someday :)

2. IPython Notebook Viewer.

I've known about this "jupyter" python thing for awhile, but I've forgot about it, and Filipe Fernandes helped me to remind myself about this cool feature!

BTW I think you should definitely check his blog.

3. IPython Notebook are available at github natively.

Basically you can since December 2015 use IPython Notebooks with native github UI, that will render your ipybn files.

Check this cool stuff about Monsoon and cosmic-ray

LL (Lessons Learned)

1. Trying to read gpx files twice will fail with coercing issue.

As I've described it in a sample file with comment, if you try to do something like this:

file_to_open = open("file.txt")

# bla bla bla you do something with it

# and then again:

reading_file = open(file_to_open)

But this time it will not be the same "coercing" issue error in traceback, but still it's the same principle of reusing an open'ed connector to file, to reopen file with it.

Solution:

Find line in which you have the failure and change it to this:

file_to_open = open("file.txt")

# bla bla bla you do something with it

# and then again:

reading_file = open("file.txt")

2. Did not make my commitments

Why I've failed to accomplish creating code ? Well... I'm in a learning curve on working alone in project and still failing at planning part.

Solution:

Start new post-blogs with plan and then stick with it.

Acknowledgements

How do I parse xml in python at stackoverflow

Pytrainer

Visualizing Strava Tracks with python

VimFX - vim plugin for firefox

What's next

1. Implementation of GPX Reader with plugin-engine.

2. Check my Issue list for Biking Endorphines project.

Pelican blog with Fabric and FTP deployment part 2/2

2017-03-22T19:30:00+01:00

FTP Deployment using ftputil for Pelican Blog.

Process of creating Fabric Deployment with FTP included:

1. Using Quickstarter for creating default pelican blog example.

2. Setting-up FTP Server for testing.

3. Adding ftp_upload task at Fabric default fabfile.py.

4. Exploring and development of ftp_upload task using ftputil and netrc as I've briefly predicted in post on 'Pelican blog with Fabric and FTP part1'

Development of FTP Fabric deployment

I've firstly tried to fight-out problem with ftp-server, but hence time limits, I had to bury investigating ftp-server itself (which meant to be a testbed only, not actual post).

Finally when I've found solution, I've researched how other's solutions looks like, but non of them were satisfying my needs - check stackoverflow link

I almost copy-pasted last solution, but made my tweaks to it that actually extends this with not only source but also target directory.

Finally solution looks like this:

def ftp_upload():
    netrc_data = netrc.netrc(env.netrc_file)
    authenticator = netrc_data.authenticators(env.ftp_host)

    def upload_dir(source, target):
        ftp_host.chdir(target)
        for dir_name, _, dir_fiels in os.walk(source):

            local = os.path.join(os.curdir, dir_name)
            local_strip = local.strip("./")
            local_for_remote = local_strip.strip(source)
            if not ftp_host.path.exists(target + local_for_remote):
                ftp_host.makedirs(target + local_for_remote)

            for file_ in os.listdir(local_strip):
                source_upload = local + "/" + file_
                target_upload = target + local_for_remote +"/" + file_
                if not os.path.isdir(source_upload):
                    logging.debug("uploads {},  {}".format(source_upload, target_upload))
                    ftp_host.upload(source_upload, target_upload)

    with ftputil.FTPHost(env.ftp_host, authenticator[0], authenticator[2]) as ftp_host:
        upload_dir(env.deploy_path, env.ftp_target_dir)

This solution uses netrc file as you see at the second line.

An example of netrc file that KEEP-IN-MIND needs to have a chmod 0600 .netrc mode, looks like this:

machine ftp.freebsd.org
    login anonymous
    password edwin@mavetju.org

You may use this solution with fabric as described below:

fab --set ftp_host=IP_ADDRESS_OR_DNS_NAME_OF_FTP,ftp_target_dir=/DIRECTORY_AT_FTP_SERVER_TO_WHICH_FILES_WILL_BE_UPLOADED/,netrc_file=/YOUR_ABSOLUTE_PATH_LOCAL_NETRC_FILE ftp_upload

i.e.

fab --set ftp_host=127.0.0.1,ftp_target_dir=/var/www/,netrc_file=/root/.netrc ftp_upload

Code commits done for this post:

Adds simplest version of fab-ftp using lftp

Adds ftp_upload task

Tools and applications used:

Docker (for ftp-server and for pelican-deployment tests)
Vi/VIM as my main editor
Tmux

LL (Lessons Learned)

1. Using docker and ftp service introduce networking problem.

Using ftp service called "pureftpd" as docker container was not a perfect solution as I've found out in a painfull manner.

I've tried to test my ftp_upload fabric task, but I could not even connect with simple ftp on second docker image nor via host itself.

I've stumbled upon error:

500 I won't open a connection to 172.17.0.155 (only to 172.17.42.1)
ftp: bind: Address already in use

Solution:

I needed to change network settings of docker container to share all network with host - with adding at docker run parameter --net=host

2. Using pureftpd as testing ftp-service introduce too-much-connections-per-user error.

I was testing putting files to ftp-server, and I could not use ftputil nor lftp command to put files. I could only use simplest ftp command to put files to server.

Error I've got:

Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 756, in main
        *args, **kwargs
    File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 426, in execute
        results['<local-only>'] = task.run(*args, **new_kwargs)
    File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 173, in run
        return self.wrapped(*args, **kwargs)
    File "/app/fabfile.py", line 120, in ftp_upload
        upload_dir(env.deploy_path, env.ftp_target_dir)
    File "/app/fabfile.py", line 117, in upload_dir
        ftp_host.upload(source_upload, target_upload)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 475, in upload
        conditional=False, callback=callback)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/file_transfer.py", line 184, in copy_file
        target_fobj = target_file.fobj()
    File "/usr/local/lib/python2.7/dist-packages/ftputil/file_transfer.py", line 94, in fobj
        return self._host.open(self.name, self.mode)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 205, in open
        host = self._copy()
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 144, in _copy
        return self.__class__(*self._args, **self._kwargs)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 71, in __init__
        self._session = self._make_session()
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 131, in _make_session
        session = factory(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/error.py", line 134, in __exit__
        raise TemporaryError(*exc_value.args, original_exception=exc_value)
ftputil.error.TemporaryError: 421 I can't accept more than 1 connections as the same user
Debugging info: ftputil 3.3.1, Python 2.7.6 (linux2)

Solution:

I tried to change configuration, set PerUserLimit's and other, but it didn't work.

Finally in my desperation and a little bit too much time spend on tracking this issue then actual development I've dropped this thing for now and changed ftp server to much less "secure" - to proftpd with predefined docker image on x64 bit - called hauptmedia/proftpd - I recommend you to check his github account projects :).

This container works properly and for now is my testbed.

When I've got more time I'll investigate problem with pureftpd and it's "security-by-obscurity" so-to-speak problem.

Acknowledgements

FTPUtil Documentation

Simple FTP Mirror - By Thimo Kraemer

What's next

1. Using pelican-bootstrap3 theme (or other), extend possibilities with adding `fancybox` as image viewer.

2. See ISSSUES task-list for this 'mini' growing project :)

See you at next Wednesday :)

Github Pages and Travis auto Deployment

2017-03-19T18:30:00+01:00

Auto Deployment of Pyreverse output to Github Pages!

That's our today's goal - to achieve it there is a lot of middle steps to take-over.

I've found a best tutorial I could - Automatically Update Github Pages with Travis

I've follow it with successful auto-updating github page.

Ofcourse I've tweaked it to my preferences and behaviour I wanted it to be doing - generate pyreverse and push it to gh-pages.

Prerequisites

Github Pages is a Github place for keeping project's documentation . It's supports Jekyll (Ruby-Based) static page generator - in addition to regular HTML content.

Pyreverse will generate png files with class-structure and package-structure(lined), we will add this to gh-pages of our project as a Jekyll markdown md file.

How to setup your own auto-generator.

Following steps are almost identical with those presented in tutorial at the beginning of the post.

1. You need to configure source of gh-pages for your github repository

That can be done within "Github Pages" section of repository settings. Template to link with settings is below:

https://github.com/{NAME_OF_USER}/{NAME_OF_REPOSITORY}/settings/

There at the Source you have drop-down list from which you can select your branch.

Be-aware - `gh-pages` branch is not created by default, by default `master` branch is used. You will need to create it manually as I did.

2. Create Github Token for travis to use it.

Generate token at THIS PAGE - You will need to re-authorize yourself again for this page.

You add name of your token and set scope for the token - I've used only for REPO - to not overuse this token in other situations, and be sure that only repositories can be harmed if it will be compromized.

Save your generated token - it will only be visible NOW

3. Setting up Travis

Once you've saved your token, you'll need it to generate secure-token for your .travis.yml configuration file.

For this part I needed to use ruby-based docker image - but this time taking into account time-pressure, I've used x64 bit for spending less time (not that I didn't tried first building x32 image for ruby ... but that's for another post).

Ruby image with docker has a gem installer that will help us install travis commandliner that we need to create secure tokens.

In image (or if you have already installled ruby with gem) :

gem install travis

That's it - super easy. Now use travis command - best to use in your project source-code that have already .travis.yml file - cause it will change the file adding secure-token to it automatically:

 travis encrypt -r username/reponame GH_TOKEN=[the token you created before] --add

4. Now let's make a bash script that will generate and deploy to gh-pages.

Here I've more-then-less used code presented in tutorial at beginning of post. Here is how it looks like now:

#!/bin/bash
rev=$(git rev-parse --short HEAD)

if [ "$TRAVIS_BRANCH" != "master" ]
then
    echo "This commit was made against the $TRAVIS_BRANCH and not the master! No deploy!"
    exit 0
fi

mkdir -p deployed_gh_pages/${rev}
cd deployed_gh_pages/${rev}

git init
git config user.name "Anselmos"
git config user.email "Github.com/Anselmos"

git remote add upstream "https://$GH_TOKEN@github.com/anselmos/Biking-Endorphines.git"
git fetch upstream
git reset upstream/gh-pages

cp ../../generated_pyreverse/* .  # you put here your files from whatever place you generated before this script

git add -A . # this will add all your files to staging mode
git commit -m "rebuild pages at ${rev}"  # this will commit all the files put in stage mode with information about commit-id - That's what I also wanted :) - you can track which commit made pyreverse
git push -q upstream HEAD:gh-pages  # push to github gh-pages branch.

Final results:

Each time new push to master will be done, travis will not only check as usually pylint/unit-test, but also will create pyrevesed output that is automatically moved to gh-page seen here:

Pyreverse Generated GH Page per each master push

Code commits done for this post:

A lot of try-error commits :
My first Github Pull Request that I'm proud of :
- Pyreverse to gh pages

Tools and applications used:

vi/vim
ruby gem installer
travis commandliner
Docker
pyreverse

LL (Lessons Learned)

1. Using separated branch for testing on travis - goes with error.

At first travis used separated branch and worked properly. When I've setup Github Pages at projects account Travis probably found that project is using GH-Pages and stopped all tasks on this branch.

When I've done second attempt to push to branch called pyreverse-to-gh-pages it didn't even start.

Then I was a little bit worried, what happen. I've started my research to find out why travis didn't even start to build the branch.

On the "menu options" at travis-ci page I've found "requests" page where I could see if request to start job on travis is actually working.

There I've found this beautiful error that at first glance meant nothing to me:

Github Pages branch not included via configuration

Then I've started to check if there is any type of error in my .travis.yml file as it was before by trying to add secure-tokens. But no luck with that.

Solution:

I've finally found out that Travis will revoke any branches that contains "gh-pages" name by default(at least that's my thinking about it). Check information here Safelisting or blocklisting branches - to overcome this issue I've changed .travis.yml settings and add following:

branches:
  only:
      - gh-pages
      - /.*/

That seem to made a trick.

2. Forgot about dependencies used when pyreverse generates outputs.

Pyreverse for proper usage needs "Graphviz" installed at system level. Since it's a system dependency, it's not included at requirements.txt level.

Solution:

Add installation of Graphviz at travis configuration.

Acknowledgements

Auto-deploying built products to gh-pages with Travis

Using Github Pages to host your website

Update Github Pages with Travis tutorial

What's next

1. I'll finally start coding on GPX File reader - see at Issue #7

2. Check my TODO-LIST/Issues at Github

Modelling tools for software - Pyreverse for Biking Endorphines

2017-03-18T00:00:00+01:00

Modelling tools and their capabilities

Why would you even bother using modelling tools and technologies? :

1. Using modelling tools keeps track of how your classes/flow/environment will look like and how they will work with each-other.

Check information from University of Cambridge about Software Design - course materials.

2. Designing software will push you to think-through about specific components in project.

You will probably think about elements in project, but human-beings have more visual-memory developed, hence I'll present you something:

3. Creating complex software-engineering system needs some documentation and proper design.

Think about NASA - if they would not design and keep documentation, how would they be able to comunicate with engineers in space and help them with problems they are facing day-by-day ?

Plan for the spacecraft that was designed at 21 May 1959:

4. You can reverse process and instead of using UML only for modelling, use it for creating source code.

In my professional experience I've used DIA some time ago. You can even read content of the dia diagrams - because they are simple xml files.

DIA is the modelling tool that with extension called dia2code( dia2code on github) can be used to generate code based on UML design.

Ofcourse you may be sure that dia2code will only reshape your diagrams into simplest classes with methods, but you will need to fill it with some juice-source-code

Why use reverse-engineering tool like pyreverse ?

1. You need to investigate a source code that you haven't created and code has more then 10 classes and relationships between them.

I.e. You get involved in new project. Projects has already some code that is hard to read and understand in first hand. You may find yourself reading and analysing source code and creating flow blocks how code works, that reverse-engineering tool could do for you in almost snap-of-a-finger.

Making example in real-life - you want to know what are pelican source-code relations to better understand how it works:

git clone https://github.com/getpelican/pelican
pyreverse -o png -ASmy pelican/pelican/

Output is big - it contains all classes , it's methods and relationships, hence it's approximately 2MB.

Pyreverse on Pelican - PNG File ~2MB

Let's make a more sliced up example, previous version contained test classes that let's say for this example don't have particular usage.

So how to ignore this files ?

Check asciinema video:

Tests are located at directory pelican/pelican/tests, hence to ignore them, we needed to add --ignore tests.

You can use --ignore multiple times. Take only into account that you need to use relative path - instead of i.e. pelican/pelican/tests - simply tests

2. Want to have visual presentation of what has been already created.

You want to see how your classes and methods looks like outside of programming language context ??

Check reverse engineering tools to read your classes from source-code.

Prerequisites

First we need to add dependencies to our run environment to make pyreverse work properly.

Pyreverse has been integrated to pylint from 0.5.1 version.

Biking-Endorphines already has pylint integrated in requirements.txt, hence for this project I will not need to install it.

If you would need to integrate pyreverse to your project, here are instructions:

apt-get install -y graphviz 
apt-get install -y python-pip
pip install pylint

Pyreverse on Biking Endorphines.

1. Normal usage, see example at asciinema!

Result (for web-project):

2. Specific class only:

But you can also do a specific class research:

pyreverse -c User -o png bikingendorphines/web/models.py

Result?

Code commits done for this post:

Since I still haven't figure out where to place pyreverse outputs and want to have them generated whenever travis will succeed - no commits for this post.

Tools and applications used:

Vi/VIM
Docker
Pyreverse
Asciinema

LL (Lessons Learned)

1. Different versions of Pyreverse at Internet.

This is like the first time I actually use pyreverse - I've only heard about it or use it like once couple of years ago.

Now as it turns out there are unmaintained version of pyreverse of 0.5.1 version, and the integrated with pylint version.

There is no such thing like proper documentation in the Internet. There is only man-page documentation and the helper-info from pyreverse command itself.

You may also search for usage of this tool by others.

Solution:

Read carefully manual page or helper info from pyreverse --help, ask me (not that I'm experienced with it or something :) ) or search for solution among other developers/technicians.

2. Pyreverse --ignore was a total nightmare.

I've thought I'll use either an absolute path or relative path (but i.e. of pelican : pelican/pelican/tests) - No luck with that.

Finally I've found out HERE, that the path is relative, but starts from where you add source-code to look for (ahh should be more carefull about that ).

Acknowledgements

Exploring Pyreverse at pythonhosted.org

Generate Python code using UML at StackOverflow

Generate UML for python source-code at StackOverflow

List of UML Tools at wiki

PlantUML

Visual Memory

NASA news

Quora- How Could I get a spaceship designed by NASA

Nasa Graphics Manual January 1976

What's next

1. Find a way to make travis push pyreverse output on successful master build - either at my blog or at github - maybe github-pages?

2. Check my TODO-LIST/Issues at Github

Pelican blog with Fabric and FTP deployment part 1/2

2017-03-15T23:00:00+01:00

FTP at Fabric - Mini Project - Requirements

There are few other attempts to create ftp connection using fabric. I've found most of them interesting. The most closest to my version can be found How I Deployed a PHP App via FTP using Fabric and Git

What Ftp libraries for python there are ?

Main thing that I'm interested in is support of netrc file - where I put credentials.

I've found examples that can be used for .netrc file with pycurl:

I know that using PyCurl can be a good option in this situation, but for now I'm dropping this solution.

FTP pre-solution

I didn't have enough time to make best source-code for this problem, so at least I've spend that time for investigation of solutions that are available.

But a quick-dirty solution - also called "duct taping" can be found below:

Simplest and the less-time consuming would be for now either use make ftp_upload or a python-fabric-replicator that uses the same thing as in make:

def ftp_upload():
    # simplest solution for now.
    local('lftp ftp://{ftp_user}@{ftp_host} -e "mirror -R {deploy_path} {ftp_target_dir}; quit"'.format(**env))

To use this part of code at Fabric execution, use this command (uses .netrc password):

fab --set ftp_user=YOUR_FTP_USER,ftp_host=YOUR_FTP_HOST,ftp_target_dir=YOUR_FTP_TARGET_DIR ftp_upload

Examples above presents using ftp mirror command.

I could use the mirror -R INPUT_DIR OUTPUT_TARGET_DIR; quit with ftplib sendcmd - But it's almost identical solution as with lftp command itself.

Code commits done for this post:

Still to be revealed :) Upcomming github repository for this Mini-Project.

Tools and applications used:

Docker (local devserver showing pelican's post I'm currently working on)
Vi/VIM as my main editor
Tmux act as my session-protector - even if I close by mistake window with ssh to server on which I'm writing this post right now using tmux, tmux will hold session and I can reconnect to it next time and everything will be the same as before I've closed window with ssh
MobaXTerm for ssh connections.

LL (Lessons Learned)

1. FTPLib is not your best-friend.

As I've found out Python's FTPLib is just a simple connector to ftp with simplest accessing to ftp. No fancy-pancy like "put_directory" or other "put_directory_mirror_files" or something simillar.

I'm a little bit dissapointed - I had great hopes with this library.

Solution:

Find better library or other libraries that are extended via this library - ftputil looks promissing

2. SFTP at the fabric source-code was a total surprice

When first I've found that there is a SFTP connector at the Fabric source-code - thought - THATS IT !

But yeah... wait a second... It's not using ftp.... hmm why not?

Right, I did forgot it's not FTP I thought - it's SSH based file transfer, not an FTP with some additional SSL/Security.

So I was back to root-cause-analysis - if I may use that medical terminology :) (Hope no medical stuff will be angry at me because of that :D )

Solution:

I will probably use the ftputil library because it contains python-friendly naming convensions and extends ftplib much more than I could want for now.

Acknowledgements

Fabric usage for remote-create-dir via ftp on StackOverflow

Git FTP at Github.com

Fabric for WP Engine site Deployments

FTPUtil Documentation

Simple FTP Mirror - By Thimo Kraemer

What's next

1. Push source code of pelican and fabric deployment with fabric code that supports ftp using some library.

2. Add additional parameters to Fabric to be able to use different environments at the same time without having to change source-code of `fabfile.py`

See you at next Wednesday - this time I hope to make post faster then 1 hour before end of day :)

Django Unit Tests at Biking Endorphines

2017-03-12T00:00:00+01:00

Unit Tests why would you need this at all?

1. Unit tests and regression testing!

Gives you confidence of not breaking your previous features by creating new features.

2. Program some more!

When you create unit-tests you have to think-through on the source-code to be proper applied to the tests.

From my own experience I have to confess - there were times when I've changed source-code of application to support properly unit-tests.

3. Time-saver !

You code new feature, you are not sure and have no proper time to test if all of "old" features are working properly.

Then you use your previously created unit-tests and find-out if at the "unit" stage, you got something broke or not.

How to use django integrated tests?

Django integrated tests uses python unittest module. This is setup with integration to django, thanks to which to run initial tests simply:

python manage.py test

But when you find yourself creating multiple django "applications" this will not work. You may need to add something like this:

python manage.py test application_directory

In my project I've added to Makefile :

python bikingendorphines/manage.py test web

How to create unit tests that are integrated with Django Framework?

Code below show how to setup your initial first test. This is placed in "test*.py" files.

"""
Unit tests for web-application.
"""
# disabling because this import will be used in future
# pylint: disable=unused-import
from django.test import TestCase
from .models import User

class UserTestCase(TestCase):
    """
    A User-class test-case
    """

    def setUp(self):
        User.objects.create(name='Bart', weight=80, height=175)
        User.objects.create(name='Trab', weight=99, height=175)

    def test_user_bmi(self):
        """
        Tests if users has been created in db
        """
        bart = User.objects.get(name='Bart')
        trab = User.objects.get(name='Trab')
        self.assertEqual(bart.bmi(), (80/(175*175)* 10000))
        self.assertEqual(trab.bmi(), (99/(175*175)* 10000))

The User class itself I've defined in this way:

class User(models.Model):
    """
    User model for obtaining personal information about biking riders
    """
    name = models.CharField(max_length=50)
    weight = models.IntegerField(default=0)
    height = models.IntegerField(default=0)

    def __unicode__(self):
        """
        Returns User information when using str/printing
        """
        return self.name

    def bmi(self):
        """
        Body Mass Index calculator simplified to number
        """
        return (self.weight / (self.height * self.height)) * 10000

Code commits done for this post:

Adds django-unit tests to Travis

Adds Web Application and pylint fixes

Add Initial unit-tests with User db class

LL (Lessons Learned)

1. Allowed_Hosts problem.

I finally was at the stage when I wanted to see if at least the debug page for djagno will popup.

But didn't know that django developers change default behaviour of ALLOWED_HOSTS and now does not add the "star" (as all-hosts allowed)

Solution:

Since I'm using VM that is simply a server-based - I had to a change allowed_hosts variable at the bikingendorphines/bikingendorphines/settings.py file with:

ALLOWED_HOSTS = [*]

2. Pylint errors.

Pylint was not optimized as I found out at the coding stage with some django - elements i.e. objects at the Models.

Pylint issue with not recognizing "migration" django scripts and finding lint-errors. I wouldn't mind if I had to manually create this files, but they were made automatically at the python manage.py makemigrations phase.

Solution:

Added pylint_django python module using pip install pylint_django and added --load-plugins pylint_django to my Makefile task that is used by Travis to complete pylint tests.

Added migrations to .pylintrc file within section ignore.

3. Pylint and Django-unit tests errors

I had some issues when integrating together Pylint and Django-Unit tests.

Solution:

For django-unit test to properly work I had to firstly remove the __init__.py python file that describes directory as a module - at the root of the bikingendorhpines project dir.

I had to change pylint to search for particular folders to find lint-errors in web- "application"

Acknowledgements

Django Tutorial- Writing and running tests

Django Models Tutorial

Django Applications

Pylint Ignore Directory at stackoverflow

Using Pylint at Django-Based projects - landscape Blog

What's Next ??

PyReverse to create a PNG files with all classes at Django and our project.
Check what I've got to do in issues of github: ISSUES

Behind closed doors - Biking Endorphines idea.

2017-03-11T00:00:00+01:00

Biking Endorphines - story REVEALED.

Story behind the bars :)

How this project came to my mind and why I've started it?

This project came to my mind as an idea, that I could motivate myself to ride bike more offen than I do now.

Second part besides of "motivation for riding" is leveling up my technical experience in Python and Android.

I begin to think, that there are sites like "Endomondo", "Strava", but they all share almost the same thing - keeping only how much kilometers/miles you've ride a bike.

They only motivate you in particular way :

to ride bike more offen by comparing your amount of kilometers you've rode (Endomondo).
to ride bike and compare timing in particular ride-path with others (Strava).

The thing that I've always missed in this type of applications was motivation for:

"adventure" - i.e. finding new roads that you can shorten/lengthen/alternate your ride
keep-up with your training - motivation for training in particular day with particular path
how high you've rode your bike
did you ride with your beloved people.

And many others that I might not think-through yet.

I thought why not use endorphines as a badge for each of this "motivation" types and create web-site that will encourage people to collect them by riding bike in different ways.

So that's the idea, now we can go-on to more technical stuff.

Phases of project

I see at least for now 5 phases of this project.

1. Web-Application.

Main part of this project. Other components will be focused around this part.

Database using standard Django ORM with all statistics data per each gps route of biking ride.

Web-part itself will be created using standard Django MVC. Planning to create:

Endorphines badges per "motivation" type.
- each biking ride data will be analysed in search for "endorphines" pattern i.e. to gather "adventure" endorphine, algorithm will compare gps coordinates data located at database.
Biking routes statistics
Calendar with all biking rides

2. REST-API.

REST API created with CRUD in mind. Will help access Android application to data available at Web-Application.

Why not SOAP Api you may ask ? Taking into account that all data will be send/received via android phone, you have to create API that will not use a lot of data to transfer information - this disqualifies SOAP as it adds a lot of additional not-needed in this situation data.

3. Android Application.

Android Application will be focused on gathering data and some simple statistics with endorphines badges already gathered.

4. Merging phase.

This phase will be integration of all above components.

5. Maintainance.

Maintainance phase will be focused on:

tweaking endorphines gathering-data statistics manipulation
tweaking gps data gathering (if any possible)
correcting any bugs

Acknowledgements

What's next

1. Go to phase 1 and start with initial code for database

2. Create first initial unit-tests and add Travis to perform this tests in parallel with pylint tests.

Blog posts deployment using Pelican and Fabric

2017-03-08T00:00:00+01:00

Introduction - How I end-up with slappy SCP?

I've decided I'll use my old laptop (which is currently used as a media server for TV) as my main "server" for maintaining and creating content and building site on it.

I have to confess, two posts (how-to-create-own-blog && get-noticed ) were deployed by me manually using SCP and "Filezilla" client. I knew it was not best idea, but still time was ticking as I wanted to keep-up with publishing first post at 1st March.

I've got into a "shortcut" mode to make it faster. And now I see it actually makes deployment slower for next-times.

I've decided to level up and move with deployment to using Fabric - since Pelican's quickstart creates this 'fab' file with defaults.

Problems with x32 bit docker-images and Fabric.

On my old laptop I've got windows in 32bit version installed, so obvious reason was to install a 32bit ubuntu VM on it and then use docker from this VM.

Then I've stepped on problem, that almost all type of docker images out-there, supports only 64bit docker-host - this means that images like python:2.7.11 also does not work - since they are supported only on 64 bit arch. A Prove of this concept can be seen on one of VM's I'm currently using - proudly presents using AsciiNema :

Now, I could re-install windows on laptop,but since this laptop has history with battery not-keeping energy and a malfuncion of power at startup, I've decided not to go this way.

Before I figure out that I could deploy second VM with 64bit ubuntu as a docker-host (since laptop supports 64bit architecture), I've decided I'll go a longer but much more satisfying way - creating my own docker images that will support 32bit hosts!

My thinking was like that: I've got an even older laptop that supports only 32bit architecture and few stationary machines - this docker images will be usefull for future deployment.

My adventure with building 32bit docker image that will support Fabric and Pelican and other deployment/development packages has began.

Building x32 Bit Docker image with packages for Fabric

I didn't have to start from "source" - since I already had an ubuntu-trusty 32bit image that I've created some time ago using tutorial which you can find HERE.

I've used a anselmos/ubuntu32 image that is available at image

As this image is a bare-bone I had to first install a pip.

Funny, but I could not download "get-pip" from image, so I've added "wget" to docker-image. To be able to install Pip I had to install a wget by which I could download a "get-pip.py" file.

Then I had to install python since there was no-python.

After that I've realized that just "pip install Fabric" will not work, because there are linux library dependencies for Fabric like : - python-dev - libffi-dev - libssl-dev

Finally I knew Pelican python-library does not go with Markdown installed as default (reStructured is) so I had to add Markdown.

Thankfully then I was able to work on docker-image and finally we can move-on to part where we actually configure our deployment for ssh/ftp

Final version of ubuntu-trusty docker x32 bit docker image with Pelican installed and Fabric ready is pelican32

You can use docker pull anselmos/pelican32 and the image will appear at your host. Keep in mind that this image will only work on x32bit Host-machine for docker. For x64bit - just use official image for python and do pip install Pelican && pip install Fabric. You may also use image that I've prepared based on python:2.7.11 image available pelican

Configuring Deployment for Fabric using ssh/ftp.

At beginning when using pelican-quickstart script, a file called fabfile.py is created with our answers.

We can change them if defaults that we set-up have change or we made a mistake.

Setup FTP connection

Unfortunatelly as I found out you cannot use Fabric with it's default fabfile.py setup for FTP.

At the time I'm writing this post, I'll not find time to fix this, but I see there's possibility to use make.

just make ftp_upload and using Makefile setup variables it will upload files

FTP_HOST=localhost
FTP_USER=bawi
FTP_TARGET_DIR=/var/www

Docker image I'm using needed installation of this packages:

apt-get install -y make
apt-get install -y lftp

I will make an update soon for how to overcome this issue using Fabric.

Setup SSH connection

Use the production variable and change to proper values, e.g. your production hostname is "localhost" and port on which ssh works is "22" and user on this host is "root" then value should be 'root@localhost:22'

For my setup I had to add also ssh-client for docker-image and add ssh-keys to image by adding docker-volume attachment.

Keep in mind, that using ssh method requires rsync installed on your system - my docker image didn't have it so ... yeah :) I had to install it :)

And also keep in mind that default directory is /var/www. If you want to change that then change variable : dest_path = '/var/www'

What's next ?

1. I will make an update on FTP fabric issue with how to overcome this.

2. Docker usage and making your own Docker image available at hub.docker.com

What do you think about this post ? Let me know in comments below - Stay Tuned ! Soon post for Get-Noticed!

Biking Endorphines - Where to Begin?

2017-03-05T15:00:00+01:00

What Github offers us.

Since project Biking Endorphines will be maintained at open-source Github, let's check what type of tools does Github offers us.

To narrow down our research, let's just find tools available as "free" - they will all be an Continuous Integration tools which purpose will be to build && test.

1. Jenkins CI

This is a fork of Hudson (or a Hudson is fork of Jenkins) - how-ever the split for Hudson/Jenkins was, Jenkins has more maintainers and commits which means it is more-actively maintained.

Jenkins is suppported natively by github with it's available services-integration-to-project

Advantage:

You can tweak the whole build & test process within it's internals.
Support for all languages (if you can set-it-up)

Disadvantage:

Setup process takes a while.
Need to configure most of the setup with web-ui
Unfortunatelly It's not something new for me.

2. SonarQube

Link: SonarQube Integration for Github

Summary: Open Source Platform for code inspection and code quality.

This uses pull-requests and before you have code available at "master" branch, you will have to merge it with this SonarQube tool. It will automatically get pull-request and using linguistics check code quality. You may also review code and find bugs on your own.

About pull-requests

How to create your first pull-request with Github

Advantage:

More complex setup for code-quality.
Support for multilple languages

Disadvantage:

A little bit to complicated and needs more time to spend for setup than I have previously.
Only for setting up code-quality tests

I may go back to this solution in future.

3. Travis CI.

About this CI I've heard for more then 2 years from now. I've always wanted to investigate usage of this CI tool.

Advantage:

It's maintainance focuses only on one .travis.yml file that has all scripts for running scripts in it.
What ever you code in your scripts, travis will take it.

Disadvantage:

It's only hosted online, if you want to deploy it on your local server/machine (correct me if I'm wrong) - if it's only for your investigation than in my opinion you could use it.
Not free for private projects/commercial projects not visible for public - Check Prices for Commercial Usage

So for me, dispite disadvantage's - is the choice - at least for now.

Issue -bug-enhancement Tracking software integrations

1. Issue tracking software available I've seen at project service integration setup.

Redmine
Trac
Jira.

2. Issue tracking I'll be using.

In my commercial experience I've used all of this bug-tracking software, hence it's a learning process I'd like to skip this to Github possibilities that are for me still a new-thing.

Add to this, that some of this tracking software can be setup as a private servers, and this project is a open-sourced - this collapse - hence Github Issues and Projects tabs are going to be used by me in this project.

Travis - be prepared - here we go!

To start with travis configuration:

1. Create .travis.yml file with appropriate configuration for your project

language: python
python:
    - "2.7"
install: "pip install -r requirements.txt"

script: 
    - make pylint

And the Makefile:

requirements:
    pip install -r requirements.txt
pylint: requirements
    pylint bikingendorphines --rcfile=.pylintrc

Make task called pylint will be executed only when requirements task will finish properly without error - that may be unneeded since .travis has already the same command at his install task, but I wanted to make sure it will work no-matter what.

As you've noticed only one script is integrated for now - it's a simple pylint check.

To setup pylint with configuration file `.pylintrc`

pip install pylint

pylint --generate-rc-file > .pylintrc ## this will generate a standard pylintrc configuration file!

You may need to tweak your .pylintrc file for better code-quality-check

2. Add project using TravisCI Profile Site

You need to login to your account of github.com using TravisCI Profile site and then select project that you want to have automatical builds.

3. Push this file to your repository

After setting up at TravisCI site project that will use auto-builds, you can now push your first .travis.yml file to your github repository.

4. See what happens at Travis-CI status page of project.

I've prepared for the worst-case scenario and that's why I've used only pylint for now to just check how the process of integration with Travis works - still had to overcome some issues with pylint for it not to break at any-point.

You can check yourself how the process of pylint'ing at TravisCI works for my project here:

LL (Lessons Learned)

1. I have a poor time-management.

Solution:

Usage of POMODORO TECHNIQUE - Special thanks for encouragement are for Krzysztof Jonczyk and his speach about Pair Programming at Boiling Frogs

2. Configuration of Integration tools are not an easy-peasy-lemon-squeezy

It does not matter if it's a SonarQube / Trac/ TravisCI / Jenkins - all of this integration tools will take time to configure.

Issues:

Since I'm starting this project, besides of TravisCI .travis.yml I had to:
- virtualenv for pip requirements.txt file using pip freeze > requirements.txt
- install pylint using pip pip install pylint
- install framework I'll be using for for purpose of pylint check pip install Django
- create initial django-project django-admin startproject bikingendorphines for pylint to be able to work on files.
- Go to TravisCI Profile Site and FIRST enable TravisCI builds on github project you wish to have auto-builds. Then push .travis.yml file to repository to trigger automatical build.

Solution:

Estimate more time for integration tools !

3. Django initial project setup is not perfect for default values of pylint-checker configuration

Solution:

For each of new "directories" add __init__.py - so that all directories becomes "python modules"
Needed to use #pylint: disable= for:
- Default django configuration variables that are not compatible with "invalid-name" check (they are seen as constants by pylint checker - and as a constants, should be upper-case, but they are not - if you are aware why, let me know in comments below )
  - urlpatters at urls.py
  - application at wsgi.py
- Warning of importing django and not using it at source-code at manage.py in line with import django
Pylint has a checker for locally disabling line checks at code-level. Since I've used "#pylint: disable" - pylint found this lines and throw at me with error of locally-disabled - To by-pass this error - I've added locally-disabled to disable at .pylintrc

Acknowledgements

What's next

Integration of git-commits with Issues at Github
Features I'll focus in this project.
Tools section showing what tools I've used for purpose of each blog-post-session.

Get Noticed! - Programming Contest for making your own programming project and blogging about it!

2017-03-02T01:00:00+01:00

Get Noticed ??

So what's "Get Noticed" about ? It's a Polish programming Contest that covers creating your own project and making 2 posts a week.

From those 2 posts, one must be with information about Contest project, the second one can be about any topic that's connected with programming/IT/technologies or Contest project.

The Contest takes a 3-months challenge for maintaining and developing open-sourced (available at github) code and making 2 posts a week.

Involvement Time!

That's just funny, but first time I've heard about "get-noticed!" I've thought - I'm a little bit to scared to get involved in such a big-deal.

But wait a second. Do you - I mean YOU - reader of this post - have a spare lifetime?

Yeah, me neither, so instead of being a chicken I've decided to step-up.

One inspiration to starting at this contest was not having a lot of coding lately neither at work nor at home.

Second inspiration that pushed me to my limits of bravery is Wroclaw's Conference called BoilingFrogs and their lectures.

Fortunatelly for me and other contest participants, registration has been extended to 12th of March link-to-post-in-polish.

Project Code-Name :)

Project I've decided to reactivate (or perhaps to resurrect would be a better word for it - since it was almost 2 years ago) was an old idea of Biking Statistics.

Now - Biking Statistics was an idea of creating better statistics than there are for Endomondo's site, but using their input from Android application (GPX files).

It worked, but I never had time to finish it up. And so it finally was a "shelf" project - You may find it at my github account somewhere -don't hope for a lot.

The contest project name is different, it's .... fanfare's in background ....

Biking Endorphines

I'll say no more about project. Soon when idea will grow up I'll inform you :)

What's next ??

Within this 3 months I hope to give you such a nice ride that you will not dare to abandon this blog :)

But seriously - In this couple of weeks I'll proudly present you :

All of my failures,
All of my successes,
Process of creating project (all-life-cycle),
Hardware & Software Requirements,
Testing libraries/Tools used,
IDE's used,
Tools (other then IDE's) used,
Programming Language used,
Programming Language libraries used,
And much, much, more.

Stay tuned :) Soon new post :)

How to make your own Blog using Pelican Python Static-sites generator engine.

2017-03-01T00:00:00+01:00

A word of introduction before we go to 'tutorial' part. If you want to go to the tutorial part right-away - please use Making Blog using Pelican

Introduction

I've been struggling on how to begin my own blog (which post will be best) and then found out, that I'm thinking about re-creating blog-engine or using existing.

So after starting to inspect what type of engine's for blogs there are - I've got my "lightbulb" - Why not use this as a post ?

I've even see this as type of "inception" - to post about posting blog :) .

And so we begin our adventure.

Types of Blog-Engines and Questions to Blog-Readers.

There are two types of blog-engines - dynamically generated and static-html-generators

Based on what purpose and your preference, you pick one of those.

You have to ask your-self few questions to know which type of engine and blog-type you'll need.

Do you want to write your blog (almost) without using internet ?

If you want to create your posts for blog at your "spare" time and when you have time upload them to server, then definitely you should pick the static-html-generators type.

After writing article for blog using ".rst" (reStructured) or ".md" (Markdown) file, you will generate html files and push them to server.
Do you have finance for keeping server that uses some programming language (i.e. c#, java, php, python) ?

Depending on what type of language you pick, you may find yourself in a problem of too-expensive server for your usage, or you may have to use your own ISP provider with your hardware at your home. (VPS is also an option)
Do you have time for maintaining own blog-engine ?

Did you know, that the most popular blog-engine called "wordpress" has it's own vulnerability page with different types of vulnerabilities ? It means that as soon as a patch for those vulnerabilities is available, you OUGHT TO upgrade your wordpress to this "patched" version ? If you still going to use WP than please have a few minutes and get familiar with THIS LINK
Do you like Markdown/reStructured/ASCIDoc/etc?

Both static-html-generators and dynamically generated can be used for writing your posts in those languages for plain-text-formatting.
Why not to try static-generator for blog-posts ?

I admit - this blog is created using only VIM, bash terminal and Pelican python static-html-generator. I've given a try and for now it's working for me. Maybe after-while I'll change my decision ? But right-now I'm happy with how this system works. It needs some tweaks here and there, but it's more cheap-maintainable and easy-to-use then other I've used before.

But as usual - that's only my opinion - yours may vary depending on preferences.

Why Pelican ?

In case of python static-site generators, there are only few of them, that you may use. I've found Pelican best fitting my needs. Below are Dis-Advantages for Pelican:

Advantages of Pelican:

Great amount of themes that you can shuffle.
Great amount of plugins
Python expansibility.
Using both Markdown, reStructured files for writing content.
Quick-Starter script that helps creating initial setup for your blog.
Using Fabric for deployment.
Add Disqus to your Posts is super-easy - just add "DISQUS_SITENAME" and set at administration of DISQUS website on which you'll use DISQUS and off-you-go :)

Disadvantages of Pelican:

Some themes may be complicated and not explained enough for a "quick" shuffle - you may occur some errors upon changing theme.
Some Extensions need to be also set at pelicanconf instead of theme-config.

Making Blog using Pelican

1. Installation of python package:

pip install pelican
pip install markdown # Only if you intent to use Markdown

2. Setup pelican using quick-start script build-in with pelican:

pelican-quickstart

You'll answer questions and get a standard setup with name of blog and setup for publishing. If needed help, check at : Kickstart-Your-Site

3. Go to directory you setup in quick-start guide. Go to directory "content" and place a file like "my_first_post.md" with following:

Title: MyFirst Post Title
Date: 2017-03-01 17:30:30
Tags: initial, pelican

This is information that will be visible as content of this "myfirstpost" page.

4. Go back to pelican directory and :

make devserver

Which will use created pages in content directory and startup a development-type page at your localhost with port 8000 page

Using Themes

To make use of themes with Pelican you need to:

1. Clone pelican-themes repository : `git clone --recursive https://github.com/getpelican/pelican-themes /data/pelican-themes`

2. Clone pelican-plugins repository : `git clone --recursive https://github.com/getpelican/pelican-plugins /data/pelican-plugins`

3. Setup theme with it's description that should be in "README.md" or other similar file.

How to use this blog's theme ?

In this particular blog I've used pelican-bootstrap3 theme. I've picked this theme, because it supports responsiveness to changing window size(you can resize your window and content will not be cut-out in some strange way).

To setup this Theme I had to make few steps (necessary):

1. Clone thems & plugins ( as presented in #1 & #2 )

2. Add to `pelicanconf.py` :

PLUGIN_PATHS = ['/data/pelican-plugins']
PLUGINS = ['i18n_subsites']

THEME = "/data/pelican-themes/pelican-bootstrap3"
JINJA_ENVIRONMENT = {
    "extensions": ['jinja2.ext.i18n'],
}

3. Re-build blog using:

make html

Where is my content generated ??

It's at the OUTPUT_PATH which may be setup at pelicanconf.py if you set different output directory than default - meaning output/.

All Settings description can be found at settings

Thanks :)

That's all. Let me know in comments below what do you think about this blogging solution?

Do you think static-generated sites using a Pelican or other engine is your thing ?

Acknowledgements

[1] PyDanny's Choosing Blog Post.

[2] Reddit's post about blogging software comparison

[3] Python Blogging Software at python.org

[4] Pelican's Documentation Page.

[5] Pelican's Themes.

[6] Github Pelican Source-Code

[7] Nafiulis more-comprehensive post about making blog using Pelican.

[8] Markdown Cheat-Sheet at Github

What's next ?

Well ... you don't know what future may give you. I hope to solve few problems I've found while writing this post and creating my first blog using Pelican:

Anselmos Blog

Web based terminal emulators

The list

Acknowledgements

Auto-promotion

Related links

Thanks!

PYTHONSTARTUP What it is and How to use it

Examples of pythonrc

Acknowledgements

Auto-promotion

Related links

Thanks!

How to use python debug with pdb

Calling PDB Stack Trace

Using PDB - commands to move in pdb stack-trace

Acknowledgements

Auto-promotion

Related links

Thanks!

How to Automate web-scraping vocabulary into pdf

Scraping data from vocabulary.com

Saving html data into pdf

Source code

Acknowledgements

Auto-promotion

Related links

Thanks!

How to use sed as a refactor tool replacement

Acknowledgements

Auto-promotion

Related links

Thanks!

Fixing pelican_link_to_title no title problem

The issue

Fixing the problem

Checking if link is html page.

Source of plugin fixed.

Acknowledgements

Auto-promotion

Related links

Thanks!

Accessing Polar A300 data with polaraccesslink api

API

Reusing Example code

Acknowledgements

Auto-promotion

Related links

Thanks!

Ascii Easter bunny from picture

Img2Txt

Installation

Testing img2txt

MotionDesignStudio script

Fixing issues with script

Usage

Acknowledgements

Auto-promotion

Related links

Thanks!

Debugging https problem on pelican_link_to_title plugin

The failed https page link

Replacing urllib into requests.

Acknowledgements

Auto-promotion

Related links

Thanks!

How to use ack2 searching tool for code

Docker for building ack2

Ack on Docker alpine

Installation

Usage

Acknowledgements

Auto-promotion

Related links

Thanks!

Regexp vs clean python code performance check

Regexp example.

Performance regexp

Acknowledgements

The `focus` problem.