# ---
# jupyter:
#   jupytext:
#     cell_metadata_json: true
#     formats: ipynb,py:percent
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.3'
#       jupytext_version: 1.10.2
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# %% [markdown]
# # Dimensionality Reduction in [Bayer and Luetticke (2018)](https://cepr.org/active/publications/discussion_papers/dp.php?dpno=13071)
#
# [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/econ-ark/HARK/BayerLuetticke?filepath=HARK%2FBayerLuetticke%2FDCT-Copula-Illustration.ipynb)
#
# This companion to the [main notebook](TwoAsset.ipynb) explains in more detail how the authors reduce the dimensionality of their problem
#
# - Based on original slides by Christian Bayer and Ralph Luetticke
# - Original Jupyter notebook by Seungcheol Lee
# - Further edits by Chris Carroll, Tao Wang
#

# %% [markdown]
# ### Preliminaries
#
# In Steady-state Equilibrium (StE) in the model, in any given period, a consumer in state $s$ (which comprises liquid assets $m$, illiquid assets $k$, and human capital $\newcommand{hLev}{h}\hLev$) has two key choices:
# 1. To adjust ('a') or not adjust ('n') their holdings of illiquid assets $k$
# 1. Contingent on that choice, decide the level of consumption, yielding consumption functions:
#     * $c_n(s)$ - nonadjusters
#     * $c_a(s)$ - adjusters
#
# The usual envelope theorem applies here, so marginal value wrt the liquid asset equals marginal utility with respect to consumption:
# $[\frac{d v}{d m} = \frac{d u}{d c}]$.
# In practice, the authors solve their problem using the marginal value of money $\texttt{Vm} = dv/dm$, but because the marginal utility function is invertible it is trivial to recover $\texttt{c}$ from $(u^{\prime})^{-1}(\texttt{Vm} )$.  The consumption function is therefore computed from the $\texttt{Vm}$ function

# %% {"code_folding": [0]}
# Setup stuff

# This is a jupytext paired notebook that autogenerates a corresponding .py file
# which can be executed from a terminal command line via "ipython [name].py"
# But a terminal does not permit inline figures, so we need to test jupyter vs terminal
# Google "how can I check if code is executed in the ipython notebook"
def in_ipynb():
    try:
        if str(type(get_ipython())) == "<class 'ipykernel.zmqshell.ZMQInteractiveShell'>":
            return True
        else:
            return False
    except NameError:
        return False

# Determine whether to make the figures inline (for spyder or jupyter)
# vs whatever is the automatic setting that will apply if run from the terminal
if in_ipynb():
    # # matplotlib inline generates a syntax error when run from the shell
    # so do this instead
    get_ipython().run_line_magic('matplotlib', 'inline')
else:
    from matplotlib.pyplot import ion
    ion()
    get_ipython().run_line_magic('matplotlib', 'auto')

# The tools for navigating the filesystem
import sys
import os

# Find pathname to this file:
my_file_path = os.path.dirname(os.path.abspath("DCT-Copula-Illustration.ipynb"))

# Relative directory for pickled code
code_dir = os.path.join(my_file_path, "../Assets/Two")

sys.path.insert(0, code_dir)
sys.path.insert(0, my_file_path)

# %% {"code_folding": [0]}
# Load precalculated Stationary Equilibrium (StE) object EX3SS

import pickle
os.chdir(code_dir) # Go to the directory with pickled code

## EX3SS_20.p is the information in the stationary equilibrium
## (20: the number of illiquid and liquid weath gridpoints)
### The comments above are original, but it seems that there are 30 not 20 points now

EX3SS=pickle.load(open("EX3SS_20.p", "rb"))

# %% [markdown]
# ### Dimensions
#
# The imported StE solution to the problem represents the functions at a set of gridpoints of
#    * liquid assets ($n_m$ points), illiquid assets ($n_k$), and human capital ($n_h$)
#       * In the code these are $\{\texttt{nm,nk,nh}\}$
#
# So even if the grids are fairly sparse for each state variable, the total number of combinations of the idiosyncratic state gridpoints is large: $n = n_m \times n_k \times n_h$.  So, e.g., $\bar{c}$ is a set of size $n$ containing the level of consumption at each possible _combination_ of gridpoints.
#
# In the "real" micro problem, it would almost never happen that a continuous variable like $m$ would end up being exactly equal to one of the prespecified gridpoints. But the functions need to be evaluated at such non-grid points.  This is addressed by linear interpolation.  That is, if, say, the grid had $m_{8} = 40$ and $m_{9} = 50$ then and a consumer ended up with $m = 45$ then the approximation is that $\tilde{c}(45) = 0.5 \bar{c}_{8} + 0.5 \bar{c}_{9}$.
#

# %% {"code_folding": [0]}
# Show dimensions of the consumer's problem (state space)

print('c_n is of dimension: ' + str(EX3SS['mutil_c_n'].shape))
print('c_a is of dimension: ' + str(EX3SS['mutil_c_a'].shape))

print('Vk is of dimension:' + str(EX3SS['Vk'].shape))
print('Vm is of dimension:' + str(EX3SS['Vm'].shape))

print('For convenience, these are all constructed from the same exogenous grids:')
print(str(len(EX3SS['grid']['m']))+' gridpoints for liquid assets;')
print(str(len(EX3SS['grid']['k']))+' gridpoints for illiquid assets;')
print(str(len(EX3SS['grid']['h']))+' gridpoints for individual productivity.')
print('')
print('Therefore, the joint distribution is of size: ')
print(str(EX3SS['mpar']['nm'])+
      ' * '+str(EX3SS['mpar']['nk'])+
      ' * '+str(EX3SS['mpar']['nh'])+
      ' = '+ str(EX3SS['mpar']['nm']*EX3SS['mpar']['nk']*EX3SS['mpar']['nh']))

# %% [markdown]
# ### Dimension Reduction
#
# The authors use different dimensionality reduction methods for the consumer's problem and the distribution across idiosyncratic states

# %% [markdown]
# #### Representing the consumer's problem with Basis Functions
#
# The idea is to find an efficient "compressed" representation of our functions (e.g., the consumption function), which BL do using tools originally developed for image compression.  The analogy to image compression is that nearby pixels are likely to have identical or very similar colors, so we need only to find an efficient way to represent how the colors _change_ from one pixel to nearby ones.  Similarly, consumption at a given point $s_{i}$ is likely to be close to consumption point at another point $s_{j}$ that is "close" in the state space (similar wealth, income, etc), so a function that captures that similarity efficiently can preserve most of the information without keeping all of the points.
#
# Like linear interpolation, the [DCT transformation](https://en.wikipedia.org/wiki/Discrete_cosine_transform) is a method of representing a continuous function using a finite set of numbers. It uses a set of independent [basis functions](https://en.wikipedia.org/wiki/Basis_function) to do this.
#
# But it turns out that some of those basis functions are much more important than others in representing the steady-state functions. Dimension reduction is accomplished by basically ignoring all basis functions that make "small enough" contributions to the representation of the function.
#
# ##### When might this go wrong?
#
# Suppose the consumption function changes in a recession in ways that change behavior radically at some states.  Like, suppose unemployment almost never happens in steady state, but it can happen in temporary recessions.  Suppose further that, even for employed people, in a recession, _worries_ about unemployment cause many of them to prudently withdraw some of their illiquid assets -- behavior opposite of what people in the same state would be doing during expansions.  In that case, the basis functions that represented the steady state function would have had no incentive to be able to represent well the part of the space that is never seen in steady state, so any functions that might help do so might well have been dropped in the dimension reduction stage.
#
# On the whole, it seems unlikely that this kind of thing is a major problem, because the vast majority of the variation that people experience is idiosyncratic.  There is always unemployment, for example; it just moves up and down a bit with aggregate shocks, but since the experience of unemployment is in fact well represented in the steady state the method should have no trouble capturing it.
#
# Where the method might have more trouble is in representing economies in which there are multiple equilibria in which behavior is quite different.

# %% [markdown]
# #### For the distribution of agents across states: Copula
#
# The other tool the authors use is the ["copula"](https://en.wikipedia.org/wiki/Copula_(probability_theory)), which allows us to represent the distribution of people across idiosyncratic states efficiently
#
# The copula is computed from the joint distribution of states in StE and will be used to transform the [marginal distributions](https://en.wikipedia.org/wiki/Marginal_distribution) back to joint distributions.  (For an illustration of how the assumptions used when modeling asset price distributions using copulas can fail see [Salmon](https://www.wired.com/2009/02/wp-quant/))
#
#    * A copula is a representation of the joint distribution expressed using a mapping between the uniform joint CDF and the marginal distributions of the variables
#
#    * The crucial assumption is that what aggregate shocks do is to squeeze or distort the steady state distribution, but leave the rank structure of the distribution the same
#       * An example of when this might not hold is the following.  Suppose that in expansions, the people at the top of the distribution of illiquid assets (the top 1 percent, say) are also at the top 1 percent of liquid assets. But in recessions the bottom 99 percent get angry at the top 1 percent of illiquid asset holders and confiscate part of their liquid assets (the illiquid assets can't be confiscated quickly because they are illiquid). Now the people in the top 1 percent of illiquid assets might be in the _bottom_ 1 percent of liquid assets.
#
# - In this case we just need to represent how the mapping from ranks into levels of assets
#
# - This reduces the number of points for which we need to track transitions from $3600 = 30 \times 30 \times 4$ to $64 = 30+30+4$.  Or the total number of points we need to contemplate goes from $3600^2 \approx 13 $million to $64^2=4096$.

# %% {"code_folding": []}
# Get some specs about the copula, which is precomputed in the EX3SS object

print('The copula consists of two parts: gridpoints and values at those gridpoints:'+ \
      '\n gridpoints have dimensionality of '+str(EX3SS['Copula']['grid'].shape) + \
      '\n where the first element is total number of gridpoints' + \
      '\n and the second element is number of idiosyncratic state variables' + \
      '\n whose values also are of dimension of '+str(EX3SS['Copula']['value'].shape[0]) + \
      '\n each entry of which is the probability that all three of the'
      '\n state variables are below the corresponding point.')

# %% {"code_folding": [0]}
## Import BL codes

import sys

# Relative directory for BL codes
sys.path.insert(0,'../../../..')  # comment by TW: this is not the same as in TwoAsset.ipynb.
from FluctuationsTwoAsset import FluctuationsTwoAsset

# %% {"code_folding": [0]}
## Import other necessary libraries

import numpy as np
#from numpy.linalg import matrix_rank
import scipy as sc
import matplotlib.pyplot as plt

import time
import scipy.fftpack as sf  # scipy discrete fourier transforms
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.ticker import LinearLocator, FormatStrFormatter
from matplotlib import cm
from matplotlib import lines
import seaborn as sns
import copy as cp
from scipy import linalg   #linear algebra

# %% {"code_folding": []}
## Choose an aggregate shock to perturb(one of three shocks: MP, TFP, Uncertainty)

# EX3SS['par']['aggrshock']           = 'MP'
# EX3SS['par']['rhoS']    = 0.0      # Persistence of variance
# EX3SS['par']['sigmaS']  = 0.001    # STD of variance shocks

#EX3SS['par']['aggrshock']           = 'TFP'
#EX3SS['par']['rhoS']    = 0.95
#EX3SS['par']['sigmaS']  = 0.0075

EX3SS['par']['aggrshock'] = 'Uncertainty'
EX3SS['par']['rhoS'] = 0.84    # Persistence of variance
EX3SS['par']['sigmaS'] = 0.54    # STD of variance shocks

# %% {"code_folding": []}
## Choose an accuracy of approximation with DCT

### Determines number of basis functions chosen -- enough to match this accuracy
### EX3SS is precomputed steady-state pulled in above
EX3SS['par']['accuracy'] = 0.99999

# %% {"code_folding": []}
## Implement state reduction and DCT
### Do state reduction on steady state
EX3SR=FluctuationsTwoAsset(**EX3SS)   # Takes StE result as input and get ready to invoke state reduction operation
SR=EX3SR.StateReduc()           # StateReduc is operated

# %% {"code_folding": [0]}
# Measuring the effectiveness of the state reduction

print('What are the results from the state reduction?')
#print('Newly added attributes after the operation include \n'+str(set(SR.keys())-set(EX3SS.keys())))

print('\n')

print('To achieve an accuracy of '+str(EX3SS['par']['accuracy'])+'\n')

print('The dimension of the policy functions is reduced to '+str(SR['indexMUdct'].shape[0]) \
      +' from '+str(EX3SS['mpar']['nm']*EX3SS['mpar']['nk']*EX3SS['mpar']['nh'])
      )
print('The dimension of the marginal value functions is reduced to '+str(SR['indexVKdct'].shape[0]) \
      + ' from ' + str(EX3SS['Vk'].shape))
print('The total number of control variables is '+str(SR['Contr'].shape[0])+'='+str(SR['indexMUdct'].shape[0]) + \
      '+'+str(SR['indexVKdct'].shape[0])+'+ # of other macro controls')
print('\n')
print('The copula represents the joint distribution with a vector of size '+str(SR['Gamma_state'].shape) )
print('The dimension of states including exogenous state, is ' +str(SR['Xss'].shape[0]))

print('It simply stacks all grids of different\
      \n state variables regardless of their joint distributions.\
      \n This is due to the assumption that the rank order remains the same.')
print('The total number of state variables is '+str(SR['State'].shape[0]) + '='+\
     str(SR['Gamma_state'].shape[1])+'+ the number of macro states (like the interest rate)')

# %% [markdown]
# ### Graphical Illustration
#
# #### Policy/value functions
#
# Taking the consumption function as an example, we plot consumption by adjusters and non-adjusters over a range of $k$ and $m$ that encompasses 100 as well 90 percent of the mass of the distribution function,respectively.
#
# We plot the functions for the each of the 4 values of the wage $h$.
#

# %% {"code_folding": []}
## Graphical illustration

xi = EX3SS['par']['xi']
invmutil = lambda x : (1./x)**(1./xi)

### convert marginal utilities back to consumption function
mut_StE  =  EX3SS['mutil_c']
mut_n_StE = EX3SS['mutil_c_n']    # marginal utility of non-adjusters
mut_a_StE = EX3SS['mutil_c_a']   # marginal utility of adjusters

c_StE = invmutil(mut_StE)
cn_StE = invmutil(mut_n_StE)
ca_StE = invmutil(mut_a_StE)

### grid values
dim_StE = mut_StE.shape
mgrid = EX3SS['grid']['m']
kgrid = EX3SS['grid']['k']
hgrid = EX3SS['grid']['h']

# %% {"code_folding": [0]}
## Define some functions to be used next

def dct3d(x):
    x0=sf.dct(x.copy(),axis=0,norm='ortho')
    x1=sf.dct(x0.copy(),axis=1,norm='ortho')
    x2=sf.dct(x1.copy(),axis=2,norm='ortho')
    return x2

def idct3d(x):
    x2 = sf.idct(x.copy(),axis=2,norm='ortho')
    x1 = sf.idct(x2.copy(),axis=1,norm='ortho')
    x0 = sf.idct(x1.copy(),axis=0,norm='ortho')
    return x0

def DCTApprox(fullgrids,dct_index):
    dim=fullgrids.shape
    dctcoefs = dct3d(fullgrids)
    dctcoefs_rdc = np.zeros(dim)
    dctcoefs_rdc[dct_index]=dctcoefs[dct_index]
    approxgrids = idct3d(dctcoefs_rdc)
    return approxgrids

# %% [markdown]
# Depending on the accuracy level, the DCT operation choses the necessary number of basis functions used to approximate consumption function at the full grids. This is illustrated in the p31-p34 in this [slides](https://www.dropbox.com/s/46fdxh0aphazm71/presentation_method.pdf?dl=0). We show this for both 1-dimensional (m or k) or 2-dimenstional grids (m and k) in the following.

# %% {"code_folding": [0]}
## 2D graph of consumption function: c(m) fixing k and h

## list of accuracy levels
Accuracy_BL    = 0.99999 # From BL
Accuracy_Less0 = 0.999
Accuracy_Less1 = 0.99
Accuracy_Less2 = 0.95

acc_lst = np.array([Accuracy_BL,Accuracy_Less0,Accuracy_Less1,Accuracy_Less2])

## c(m) fixing k and h
fig = plt.figure(figsize=(8,8))
fig.suptitle('c at full grids and c approximated by DCT in different accuracy levels'
             '\n non-adjusters, fixing k and h',
             fontsize=(13))
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.3)

for idx in range(len(acc_lst)):
    EX3SS_cp =cp.deepcopy(EX3SS)
    EX3SS_cp['par']['accuracy'] = acc_lst[idx]
    EX3SR_cp=FluctuationsTwoAsset(**EX3SS_cp)   # Takes StE result as input and get ready to invoke state reduction operation
    SR_cp=EX3SR_cp.StateReduc()
    mut_rdc_idx_flt_cp = SR_cp['indexMUdct']
    mut_rdc_idx_cp = np.unravel_index(mut_rdc_idx_flt_cp,dim_StE,order='F')
    nb_bf_cp = len(mut_rdc_idx_cp[0])
    print(str(nb_bf_cp) +" basis functions used.")
    c_n_approx_cp = DCTApprox(cn_StE,mut_rdc_idx_cp)
    c_a_approx_cp = DCTApprox(ca_StE,mut_rdc_idx_cp)
    cn_diff_cp = c_n_approx_cp-cn_StE

    # choose the fix grid of h and k
    hgrid_fix=2  # fix level of h as an example
    kgrid_fix=10  # fix level of k as an example

    # get the corresponding c function approximated by dct
    cVec = c_a_approx_cp[:,kgrid_fix,hgrid_fix]

    ## plots
    ax = fig.add_subplot(2,2,idx+1)
    ax.plot(mgrid,cVec,label='c approximated by DCT')
    ax.plot(mgrid,ca_StE[:,kgrid_fix,hgrid_fix],'--',label='c at full grids')
    ax.plot(mgrid,cVec,'r*')
    ax.set_xlabel('m',fontsize=13)
    ax.set_ylabel(r'$c(m)$',fontsize=13)
    ax.set_title(r'accuracy=${}$'.format(acc_lst[idx]))
    ax.legend(loc=0)

# %% {"code_folding": [0]}
## 2D graph of consumption function: c(k) fixing m and h

fig = plt.figure(figsize=(8,8))
fig.suptitle('c at full grids and c approximated by DCT in different accuracy levels'
             '\n non-adjusters, fixing m and h',
             fontsize=(13))
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.3)

for idx in range(len(acc_lst)):
    EX3SS_cp =cp.deepcopy(EX3SS)
    EX3SS_cp['par']['accuracy'] = acc_lst[idx]
    EX3SR_cp=FluctuationsTwoAsset(**EX3SS_cp)   # Takes StE result as input and get ready to invoke state reduction operation
    SR_cp=EX3SR_cp.StateReduc()
    mut_rdc_idx_flt_cp= SR_cp['indexMUdct']
    mut_rdc_idx_cp = np.unravel_index(mut_rdc_idx_flt_cp,dim_StE,order='F')
    nb_bf_cp = len(mut_rdc_idx_cp[0])
    print(str(nb_bf_cp) +" basis functions used.")
    c_n_approx_cp = DCTApprox(cn_StE,mut_rdc_idx_cp)
    c_a_approx_cp = DCTApprox(ca_StE,mut_rdc_idx_cp)
    cn_diff_cp = c_n_approx_cp-cn_StE

    # choose the fix grid of h and m
    hgrid_fix=2  # fix level of h as an example
    mgrid_fix=10  # fix level of k as an example

    # get the corresponding c function approximated by dct
    cVec = c_n_approx_cp[mgrid_fix,:,hgrid_fix]

    ## plots
    ax = fig.add_subplot(2,2,idx+1)
    ax.plot(kgrid,cVec,label='c approximated by DCT')
    ax.plot(kgrid,cn_StE[mgrid_fix,:,hgrid_fix],'--',label='c at full grids')
    ax.plot(kgrid,cVec,'r*')
    ax.set_xlabel('k',fontsize=13)
    ax.set_ylabel(r'$c(k)$',fontsize=13)
    ax.set_title(r'accuracy=${}$'.format(acc_lst[idx]))
    ax.legend(loc=0)

# %% {"code_folding": []}
## Set the population density for plotting graphs

print('Input: plot the graph for bottom x (0-1) of the distribution.')
mass_pct = 0.95

print('Input:choose the accuracy level for DCT, i.e. 0.99999 in the basline of Bayer and Luetticke')
Accuracy_BS = 0.99999

# %%

# %% {"code_folding": [0]}
# Restore the solution corresponding to the original BL accuracy

EX3SS['par']['accuracy'] = Accuracy_BS
EX3SR=FluctuationsTwoAsset(**EX3SS)   # Takes StE result as input and get ready to invoke state reduction operation
SR=EX3SR.StateReduc()           # StateReduc is operated

## meshgrids for plots

mmgrid,kkgrid = np.meshgrid(mgrid,kgrid)

## indexMUdct is one dimension, needs to be unraveled to 3 dimensions
mut_rdc_idx_flt = SR['indexMUdct']
mut_rdc_idx = np.unravel_index(mut_rdc_idx_flt,dim_StE,order='F')

## Note: the following chunk of codes can be used to recover the indices of grids selected by DCT. not used here.
#nb_dct = len(mut_StE.flatten())
#mut_rdc_bool = np.zeros(nb_dct)     # boolean array of 30 x 30 x 4
#for i in range(nb_dct):
#    mut_rdc_bool[i]=i in list(SR['indexMUdct'])
#mut_rdc_bool_3d = (mut_rdc_bool==1).reshape(dim_StE)
#mut_rdc_mask_3d = (mut_rdc_bool).reshape(dim_StE)

## For BL accuracy level, get dct compressed c functions at all grids

c_n_approx = DCTApprox(cn_StE,mut_rdc_idx)
c_a_approx = DCTApprox(ca_StE,mut_rdc_idx)

# Get the joint distribution calculated elsewhere

joint_distr =  EX3SS['joint_distr']

# %%

# %%

# %% {"code_folding": [0]}
## Functions used to plot consumption functions at the trimmed grids

def WhereToTrim2d(joint_distr,mass_pct):
    """
    parameters
    -----------
    marginal1: marginal pdf in the 1st dimension
    marginal2: marginal pdf in the 2nd dimension
    mass_pct: bottom percentile to keep

    returns
    ----------
    trim1_idx: idx for trimming in the 1s dimension
    trim2_idx: idx for trimming in the 1s dimension
    """

    marginal1 = joint_distr.sum(axis=0)
    marginal2 = joint_distr.sum(axis=1)
    ## this can handle cases where the joint_distr itself is a marginal distr from 3d,
    ##   i.e. marginal.cumsum().max() =\= 1
    trim1_idx = (np.abs(marginal1.cumsum()-mass_pct*marginal1.cumsum().max())).argmin()
    trim2_idx = (np.abs(marginal2.cumsum()-mass_pct*marginal2.cumsum().max())).argmin()
    return trim1_idx,trim2_idx

def TrimMesh2d(grids1,grids2,trim1_idx,trim2_idx,drop=True):
    if drop ==True:
        grids_trim1 = grids1.copy()
        grids_trim2 = grids2.copy()
        grids_trim1=grids_trim1[:trim1_idx]
        grids_trim2=grids_trim2[:trim2_idx]
        grids1_trimmesh, grids2_trimmesh = np.meshgrid(grids_trim1,grids_trim2)
    else:
        grids_trim1 = grids1.copy()
        grids_trim2 = grids2.copy()
        grids_trim1[trim1_idx:]=np.nan
        grids_trim2[trim2_idx:]=np.nan
        grids1_trimmesh, grids2_trimmesh = np.meshgrid(grids_trim1,grids_trim2)

    return grids1_trimmesh,grids2_trimmesh

# %%

# %%

# %% {"code_folding": [0]}
## Other configurations for plotting

distr_min = 0
distr_max = np.nanmax(joint_distr)
fontsize_lg = 13

## lower bound for grid
mmin = np.nanmin(mgrid)
kmin = np.nanmin(kgrid)

# %%

# %%

# %% {"code_folding": [0]}
# For non-adjusters: 3D surface plots of consumption function at full grids and approximated by DCT
##    at all grids and grids after dct first for non-adjusters and then for adjusters

fig = plt.figure(figsize=(14,14))
fig.suptitle('Consumption of non-adjusters at grid points of m and k \n where ' +str(int(mass_pct*100))+ ' % of the agents are distributed \n (for each h)',
             fontsize=(fontsize_lg))
for hgrid_id in range(EX3SS['mpar']['nh']):

    ## get the grids and distr for fixed h
    hgrid_fix = hgrid_id
    distr_fix = joint_distr[:,:,hgrid_fix]
    c_n_approx_fix = c_n_approx[:,:,hgrid_fix]
    c_n_StE_fix = cn_StE[:,:,hgrid_fix]

    ## additions to the above cell
    ## for each h grid, take the 90% mass of m and k as the maximum of the m and k axis
    mk_marginal = joint_distr[:,:,hgrid_fix]
    mmax_idx, kmax_idx = WhereToTrim2d(mk_marginal,mass_pct)
    mmax, kmax = mgrid[mmax_idx],kgrid[kmax_idx]
    mmgrid_trim,kkgrid_trim = TrimMesh2d(mgrid,kgrid,mmax_idx,kmax_idx)

    c_n_approx_trim = c_n_approx_fix.copy()
    c_n_approx_trim = c_n_approx_trim[:kmax_idx:,:mmax_idx]  # the dimension is transposed for meshgrid.
    distr_fix_trim = distr_fix.copy()

    cn_StE_trim = c_n_StE_fix.copy()
    cn_StE_trim = cn_StE_trim[:kmax_idx,:mmax_idx]
    distr_fix_trim = distr_fix_trim[:kmax_idx,:mmax_idx]

    ## find the maximum z
    zmax = np.nanmax(c_n_approx_trim)

    ## plots
    ax = fig.add_subplot(2,2,hgrid_id+1, projection='3d')
    scatter = ax.scatter(mmgrid_trim,kkgrid_trim,cn_StE_trim,
               marker='v',
               color='red')
    surface = ax.plot_surface(mmgrid_trim,kkgrid_trim,c_n_approx_trim,
                    cmap='Blues')
    fake2Dline = lines.Line2D([0],[0],
                              linestyle="none",
                              c='b',
                              marker='o') # fake line for making the legend for surface

    ax.contourf(mmgrid_trim,kkgrid_trim,distr_fix_trim,
                zdir='z',
                offset=np.min(distr_fix_trim),
                cmap=cm.YlOrRd,
                vmin=distr_min,
                vmax=distr_max)
    fake2Dline2 = lines.Line2D([0],[0],
                              linestyle="none",
                              c='orange',
                              marker='o') # fakeline for making the legend for surface

    ax.set_xlabel('m',fontsize=fontsize_lg)
    ax.set_ylabel('k',fontsize=fontsize_lg)
    ax.set_zlabel(r'$c_n(m,k)$',fontsize=fontsize_lg)
    #ax.set_xlim([mmin,mmax])
    ax.set_ylim([kmax,kmin])
    ax.set_zlim([0,zmax])
    ax.set_title(r'$h({})$'.format(hgrid_fix))
    plt.gca().invert_xaxis()
    #plt.gca().invert_yaxis()
    ax.view_init(20, 70)
    ax.legend([scatter,fake2Dline,fake2Dline2],
              ['Full-grid c','Approximated c','Joint distribution'],
              loc=0)

# %%

# %%

# %% {"code_folding": [0]}
# For adjusters: 3D surface plots of consumption function at full grids and approximated by DCT

fig = plt.figure(figsize=(14,14))
fig.suptitle('Consumption of adjusters at grid points of m and k \n where ' +str(int(mass_pct*100))+ '% of agents are distributed  \n (for each h)',
             fontsize=(fontsize_lg))
for hgrid_id in range(EX3SS['mpar']['nh']):

    ## get the grids and distr for fixed h
    hgrid_fix=hgrid_id
    c_a_StE_fix = ca_StE[:,:,hgrid_fix]
    c_a_approx_fix = c_a_approx[:,:,hgrid_fix]
    distr_fix = joint_distr[:,:,hgrid_fix]

    ## additions to the above cell
    ## for each h grid, take the 90% mass of m and k as the maximum of the m and k axis
    mk_marginal = joint_distr[:,:,hgrid_fix]
    mmax_idx, kmax_idx = WhereToTrim2d(mk_marginal,mass_pct)
    mmax, kmax = mgrid[mmax_idx],kgrid[kmax_idx]
    mmgrid_trim,kkgrid_trim = TrimMesh2d(mgrid,kgrid,mmax_idx,kmax_idx)
    c_a_approx_trim =c_a_approx_fix.copy()
    c_a_approx_trim  = c_a_approx_trim[:kmax_idx,:mmax_idx]
    distr_fix_trim = distr_fix.copy()
    ca_StE_trim =c_a_StE_fix.copy()
    ca_StE_trim = ca_StE_trim[:kmax_idx,:mmax_idx]
    distr_fix_trim = distr_fix_trim[:kmax_idx,:mmax_idx]

    # get the maximum z
    zmax = np.nanmax(c_a_approx_trim)

    ## plots
    ax = fig.add_subplot(2,2,hgrid_id+1, projection='3d')
    ax.scatter(mmgrid_trim,kkgrid_trim,ca_StE_trim,marker='v',color='red',
                    label='full-grid c:adjuster')
    ax.plot_surface(mmgrid_trim,kkgrid_trim,c_a_approx_trim,cmap='Blues',
               label='approximated c: adjuster')
    fake2Dline = lines.Line2D([0],[0],
                              linestyle="none",
                              c='b',
                              marker='o') # fake line for making the legend for surface
    ax.contourf(mmgrid_trim,kkgrid_trim,distr_fix_trim,
                zdir='z',
                offset=np.min(distr_fix_trim),
                cmap=cm.YlOrRd,
                vmin=distr_min,
                vmax=distr_max)
    fake2Dline2 = lines.Line2D([0],[0],
                              linestyle="none",
                              c='orange',
                              marker='o') # fakeline for making the legend for surface
    ax.set_xlabel('m',fontsize=fontsize_lg)
    ax.set_ylabel('k',fontsize=fontsize_lg)
    ax.set_zlabel(r'$c_a(m,k)$',fontsize=fontsize_lg)
    #ax.set_xlim([mmin,mmax])
    ax.set_ylim([kmax,kmin])
    plt.gca().invert_xaxis()
    #plt.gca().invert_yaxis()
    ax.set_zlim([0,zmax])
    ax.set_title(r'$h({})$'.format(hgrid_fix))
    ax.view_init(20, 70)
    ax.legend([scatter,fake2Dline,fake2Dline2],
              ['Full-grid c','Approx c','Joint distribution'],
              loc=0)

# %%

# %%

# %% {"code_folding": []}
## 3D scatter plots of the difference of full-grid c and approximated c for non-adjusters

fig = plt.figure(figsize=(14,14))
fig.suptitle('Approximation errors of non-adjusters at grid points of m and k \n where ' +str(int(mass_pct*100))+ '% of agents are distributed \n (for each h)',
             fontsize=(fontsize_lg))
for hgrid_id in range(EX3SS['mpar']['nh']):

    ## get the grids and distr for fixed h
    hgrid_fix = hgrid_id
    cn_diff = c_n_approx-cn_StE
    cn_diff_fix = cn_diff[:,:,hgrid_fix]
    distr_fix = joint_distr[:,:,hgrid_fix]

    ## additions to the above cell
    ## for each h grid, take the 90% mass of m and k as the maximum of the m and k axis
    mk_marginal = joint_distr[:,:,hgrid_fix]
    mmax_idx, kmax_idx = WhereToTrim2d(mk_marginal,mass_pct)
    mmax, kmax = mgrid[mmax_idx],kgrid[kmax_idx]
    mmgrid_trim,kkgrid_trim = TrimMesh2d(mgrid,kgrid,mmax_idx,kmax_idx)
    c_n_diff_trim = cn_diff_fix.copy()
    c_n_diff_trim = c_n_diff_trim[:kmax_idx,:mmax_idx]  # first k and then m because c is is nk x nm
    distr_fix_trim = distr_fix.copy()
    distr_fix_trim = distr_fix_trim[:kmax_idx,:mmax_idx]

    ## plots
    ax = fig.add_subplot(2,2,hgrid_id+1, projection='3d')

    ax.plot_surface(mmgrid_trim,kkgrid_trim,c_n_diff_trim,
                    rstride=1,
                    cstride=1,
                    cmap=cm.coolwarm,
                    edgecolor='none')
    fake2Dline_pos = lines.Line2D([0],[0],
                              linestyle="none",
                              c='r',
                              marker='o') # fakeline for making the legend for surface
    fake2Dline_neg = lines.Line2D([0],[0],
                              linestyle="none",
                              c='b',
                              marker='o') # fakeline for making the legend for surface
    ax.contourf(mmgrid_trim,kkgrid_trim,distr_fix_trim,
                zdir='z',
                offset=np.min(c_n_diff_trim),
                cmap=cm.YlOrRd,
                vmin=distr_min,
                vmax=distr_max)
    fake2Dline2 = lines.Line2D([0],[0],
                              linestyle="none",
                              c='orange',
                              marker='o') # fakeline for making the legend for contour
    ax.set_xlabel('m',fontsize=fontsize_lg)
    ax.set_ylabel('k',fontsize=fontsize_lg)
    ax.set_zlabel(r'$c_a(m,k)$',fontsize=fontsize_lg)
    #ax.set_xlim([mmin,mmax])
    ax.set_ylim([kmax,kmin])
    plt.gca().invert_xaxis()
    #plt.gca().invert_yaxis()
    ax.set_title(r'$h({})$'.format(hgrid_fix))
    ax.view_init(20, 40)
    ax.legend([fake2Dline_pos,fake2Dline_neg,fake2Dline2],
              ['Positive approx errors','Negative approx errors','Joint distribution'],
              loc=0)

# %%

# %%

# %% {"code_folding": [0]}
# Difference of full-grid c and DCT compressed c for each level of accuracy

fig = plt.figure(figsize=(14,14))
fig.suptitle('Approximation errors in different levels of accuracy \n where ' +str(int(mass_pct*100))+ '% of agents are distributed \n (non-adjusters)',
             fontsize=(fontsize_lg))

for idx in range(len(acc_lst)):
    EX3SS_cp =cp.deepcopy(EX3SS)
    EX3SS_cp['par']['accuracy'] = acc_lst[idx]
    EX3SR_cp=FluctuationsTwoAsset(**EX3SS_cp)   # Takes StE result as input and get ready to invoke state reduction operation
    SR_cp=EX3SR_cp.StateReduc()
    mut_rdc_idx_flt_cp = SR_cp['indexMUdct']
    mut_rdc_idx_cp = np.unravel_index(mut_rdc_idx_flt_cp,dim_StE,order='F')
    nb_bf_cp = len(mut_rdc_idx_cp[0])
    print(str(nb_bf_cp) +" basis functions used.")
    c_n_approx_cp = DCTApprox(cn_StE,mut_rdc_idx_cp)
    cn_diff_cp = c_n_approx_cp-cn_StE

    hgrid_fix=1  # fix level of h as an example
    c_n_diff_cp_fix = cn_diff_cp[:,:,hgrid_fix]
    distr_fix = joint_distr[:,:,hgrid_fix]

    ## for each h grid, take the 90% mass of m and k as the maximum of the m and k axis
    mk_marginal = joint_distr[:,:,hgrid_fix]
    mmax_idx, kmax_idx = WhereToTrim2d(mk_marginal,mass_pct)
    mmax, kmax = mgrid[mmax_idx],kgrid[kmax_idx]
    mmgrid_trim,kkgrid_trim = TrimMesh2d(mgrid,kgrid,mmax_idx,kmax_idx)
    c_n_diff_cp_trim = c_n_diff_cp_fix.copy()
    c_n_diff_cp_trim = c_n_diff_cp_trim[:kmax_idx:,:mmax_idx]
    distr_fix_trim = distr_fix.copy()
    distr_fix_trim = distr_fix_trim[:kmax_idx,:mmax_idx]

    ## plots
    ax = fig.add_subplot(2,2,idx+1, projection='3d')
    ax.plot_surface(mmgrid_trim,kkgrid_trim,c_n_diff_cp_trim,
                    rstride=1,
                    cstride=1,
                    cmap=cm.coolwarm,
                    edgecolor='none')
    fake2Dline_pos = lines.Line2D([0],[0],
                              linestyle="none",
                              c='r',
                              marker='o') # fakeline for making the legend for surface
    fake2Dline_neg = lines.Line2D([0],[0],
                              linestyle="none",
                              c='b',
                              marker='o') # fakeline for making the legend for surface
    dst_contour = ax.contourf(mmgrid_trim,kkgrid_trim,distr_fix_trim,
                              zdir='z',
                              offset=np.min(-2),
                              cmap=cm.YlOrRd,
                              vmin=distr_min,
                              vmax=distr_max)
    fake2Dline2 = lines.Line2D([0],[0],
                              linestyle="none",
                              c='orange',
                              marker='o') # fakeline for making the legend for contour
    ax.set_xlabel('m',fontsize=13)
    ax.set_ylabel('k',fontsize=13)
    ax.set_zlabel('Difference of c functions',fontsize=13)
    #ax.set_xlim([mmin,mmax])
    ax.set_ylim([kmax,kmin])
    plt.gca().invert_xaxis()
    #plt.gca().invert_yaxis()
    ax.set_zlim([-2,2])  # these are magic numbers. need to fix
    ax.set_title(r'accuracy=${}$'.format(acc_lst[idx]))
    ax.view_init(10, 60)
    ax.legend([fake2Dline_pos,fake2Dline_neg,fake2Dline2],
              ['+ approx errors','- approx errors','Joint distribution'],
              loc=0)

# %%

# %%

# %% {"code_folding": [0]}
# Difference of full-grid c and DCT compressed c for difference levels of accuracy

fig = plt.figure(figsize=(14,14))
fig.suptitle('Differences of approximation errors between adjusters/non-adjusters \n where ' +str(int(mass_pct*100))+ '% of agents are distributed \n in different accuracy levels',
             fontsize=(fontsize_lg))

for idx in range(len(acc_lst)):
    EX3SS_cp =cp.deepcopy(EX3SS)
    EX3SS_cp['par']['accuracy'] = acc_lst[idx]
    EX3SR_cp=FluctuationsTwoAsset(**EX3SS_cp)   # Takes StE result as input and get ready to invoke state reduction operation
    SR_cp=EX3SR_cp.StateReduc()
    mut_rdc_idx_flt_cp = SR_cp['indexMUdct']
    mut_rdc_idx_cp = np.unravel_index(mut_rdc_idx_flt_cp,dim_StE,order='F')
    nb_bf_cp = len(mut_rdc_idx_cp[0])
    print(str(nb_bf_cp) +" basis functions used.")
    c_n_approx_cp = DCTApprox(cn_StE,mut_rdc_idx_cp)
    c_a_approx_cp = DCTApprox(ca_StE,mut_rdc_idx_cp)
    cn_diff_cp = c_n_approx_cp-cn_StE
    ca_diff_cp = c_a_approx_cp-ca_StE
    c_diff_cp_apx_error = ca_diff_cp - cn_diff_cp

    hgrid_fix=1  # fix level of h as an example
    c_diff_cp_apx_error_fix = c_diff_cp_apx_error[:,:,hgrid_fix]
    distr_fix = joint_distr[:,:,hgrid_fix]

    ## additions to the above cell
    ## for each h grid, take the 90% mass of m and k as the maximum of the m and k axis
    mk_marginal = joint_distr[:,:,hgrid_fix]
    mmax_idx, kmax_idx = WhereToTrim2d(mk_marginal,mass_pct)
    mmax, kmax = mgrid[mmax_idx],kgrid[kmax_idx]
    mmgrid_trim,kkgrid_trim = TrimMesh2d(mgrid,kgrid,mmax_idx,kmax_idx)
    c_diff_cp_apx_error_trim = c_diff_cp_apx_error_fix.copy()
    c_diff_cp_apx_error_trim = c_diff_cp_apx_error_trim[:kmax_idx,:mmax_idx]
    distr_fix_trim = distr_fix.copy()
    distr_fix_trim = distr_fix_trim[:kmax_idx,:mmax_idx]

    ## get the scale
    zmin = np.nanmin(c_diff_cp_apx_error)
    zmax = np.nanmax(c_diff_cp_apx_error)

    ## plots
    ax = fig.add_subplot(2,2,idx+1, projection='3d')
    ax.plot_surface(mmgrid_trim,kkgrid_trim,c_diff_cp_apx_error_trim,
                    rstride=1,
                    cstride=1,
                    cmap=cm.coolwarm,
                    edgecolor='none',
                    label='Difference of full-grid and approximated consumption functions')
    fake2Dline_pos = lines.Line2D([0],[0],
                              linestyle="none",
                              c='r',
                              marker='o') # fakeline for making the legend for surface
    fake2Dline_neg = lines.Line2D([0],[0],
                              linestyle="none",
                              c='b',
                              marker='o') # fakeline for making the legend for surface
    ax.contourf(mmgrid_trim,kkgrid_trim,distr_fix_trim,
                zdir='z',
                offset=np.min(-0.2),
                cmap=cm.YlOrRd,
                vmin=distr_min,
                vmax=distr_max)
    fake2Dline2 = lines.Line2D([0],[0],
                              linestyle="none",
                              c='orange',
                              marker='o') # fakeline for making the legend for contour
    ax.set_xlabel('m',fontsize=fontsize_lg)
    ax.set_ylabel('k',fontsize=fontsize_lg)
    ax.set_zlabel('Difference of approximation errors',fontsize=fontsize_lg)
    #ax.set_xlim([mmin,mmax])
    ax.set_ylim([kmax,kmin])
    plt.gca().invert_xaxis()
    #plt.gca().invert_yaxis()
    ax.set_zlim([-0.2,0.2]) # these are magic numbers. need to fix
    ax.set_title(r'accuracy=${}$'.format(acc_lst[idx]))
    ax.view_init(10, 60)
    ax.legend([fake2Dline_pos,fake2Dline_neg,fake2Dline2],
              ['+ diff','- diff','Joint distribution'],
              loc=0)

# %%

# %% [markdown]
# ##### Observation
#
# - For a given grid value of productivity, the remaining grid points after DCT to represent the whole consumption function are concentrated in low values of $k$ and $m$. This is because the slopes of the surfaces of marginal utility are changing the most in these regions.  For larger values of $k$ and $m$ the functions become smooth and only slightly concave, so they can be represented by many fewer points
# - For different grid values of productivity (2 sub plots), the numbers of grid points in the DCT operation differ. From the lowest to highest values of productivity, there are 78, 33, 25 and 18 grid points, respectively. They add up to the total number of gridpoints of 154 after DCT operation, as we noted above for marginal utility function.

# %% [markdown]
# #### Distribution of states
#
# - We first plot the distribution of $k$ fixing $m$ and $h$. Next, we plot the joint distribution of $m$ and $k$ only fixing $h$ in 3-dimenstional space.
# - The joint-distribution can be represented by marginal distributions of $m$, $k$ and $h$ and a copula that describes the correlation between the three states. The former is straightfoward. We plot the copula only. The copula is essentially a multivariate cummulative distribution function where each marginal is uniform. (Translation from the uniform to the appropriate nonuniform distribution is handled at a separate stage).
#

# %% {"code_folding": [0]}
### Marginalize along h grids

joint_distr =  EX3SS['joint_distr']
joint_distr_km = EX3SS['joint_distr'].sum(axis=2)

### Plot distributions in 2 dimensional graph

fig = plt.figure(figsize=(10,10))
plt.suptitle('Marginal distribution of k at different m \n(for each h)')

for hgrid_id in range(EX3SS['mpar']['nh']):
    ax = plt.subplot(2,2,hgrid_id+1)
    ax.set_title(r'$h({})$'.format(hgrid_id))
    ax.set_xlabel('k',size=fontsize_lg)
    for id in range(EX3SS['mpar']['nm']):
        ax.plot(kgrid,joint_distr[id,:,hgrid_id])

# %%

# %% {"code_folding": [0]}
## Plot joint distribution of k and m in 3d graph
#for only 90 percent of the distributions

fig = plt.figure(figsize=(14,14))
fig.suptitle('Joint distribution of m and k \n where ' +str(int(mass_pct*100))+ '% agents are distributed \n(for each h)',
             fontsize=(fontsize_lg))

for hgrid_id in range(EX3SS['mpar']['nh']):

    ## get the distr for fixed h
    hgrid_fix = hgrid_id
    joint_km = joint_distr[:,:,hgrid_fix]

    ## additions to the above cell
    ## for each h grid, take the 90% mass of m and k as the maximum of the m and k axis
    mk_marginal = joint_distr[:,:,hgrid_fix]
    mmax_idx, kmax_idx = WhereToTrim2d(mk_marginal,mass_pct)
    mmax, kmax = mgrid[mmax_idx],kgrid[kmax_idx]
    mmgrid_trim,kkgrid_trim = TrimMesh2d(mgrid,kgrid,mmax_idx,kmax_idx)
    joint_km_trim = joint_km.copy()
    joint_km_trim  = joint_km_trim[:kmax_idx,:mmax_idx]

    # get the maximum z
    zmax = np.nanmax(joint_distr)

    ## plots
    ax = fig.add_subplot(2,2,hgrid_id+1, projection='3d')
    ax.plot_surface(mmgrid_trim,kkgrid_trim,joint_km_trim,
                    rstride=1,
                    cstride=1,
                    cmap=cm.YlOrRd,
                    edgecolor='none',
                    vmin=distr_min,
                    vmax=distr_max)
    fake2Dline = lines.Line2D([0],[0],
                              linestyle="none",
                              c='orange',
                              marker='o') # fakeline for making the legend for contour
    ax.set_xlabel('m',fontsize=fontsize_lg)
    ax.set_ylabel('k',fontsize=fontsize_lg)
    ax.set_zlabel('Probability',fontsize=fontsize_lg)
    ax.set_title(r'$h({})$'.format(hgrid_id))
    #ax.set_xlim([mmin,mmax])
    ax.set_ylim([kmax,kmin])
    ax.set_zlim([0,zmax])
    plt.gca().invert_xaxis()
    #plt.gca().invert_yaxis()
    ax.view_init(20, 60)
    ax.legend([fake2Dline],
              ['joint distribution'],
              loc=0)

# %%

# %% [markdown]
# Notice the CDFs in StE copula have 4 modes, corresponding to the number of $h$ gridpoints. Each of the four parts of the cdf is a joint-distribution of $m$ and $k$.  It can be presented in 3-dimensional graph as below.

# %% {"code_folding": []}
## Plot the copula
# same plot as above for only 90 percent of the distributions

cdf=EX3SS['Copula']['value'].reshape(4,30,30)   # important: 4,30,30 not 30,30,4?

fig = plt.figure(figsize=(14,14))
fig.suptitle('Copula of m and k \n where ' +str(int(mass_pct*100))+ '% agents are distributed \n(for each h)',
             fontsize=(fontsize_lg))
for hgrid_id in range(EX3SS['mpar']['nh']):

    hgrid_fix = hgrid_id
    cdf_fix  = cdf[hgrid_fix,:,:]

    ## additions to the above cell
    ## for each h grid, take the 90% mass of m and k as the maximum of the m and k axis
    mk_marginal = joint_distr[:,:,hgrid_fix]
    mmax_idx, kmax_idx = WhereToTrim2d(mk_marginal,mass_pct)
    mmax, kmax = mgrid[mmax_idx],kgrid[kmax_idx]
    mmgrid_trim,kkgrid_trim = TrimMesh2d(mgrid,kgrid,mmax_idx,kmax_idx)
    cdf_fix_trim = cdf_fix.copy()
    cdf_fix_trim  = cdf_fix_trim[:kmax_idx,:mmax_idx]

    ## plots
    ax = fig.add_subplot(2,2,hgrid_id+1, projection='3d')
    ax.plot_surface(mmgrid_trim,kkgrid_trim,cdf_fix_trim,
                    rstride=1,
                    cstride=1,
                    cmap =cm.Greens,
                    edgecolor='None')
    fake2Dline = lines.Line2D([0],[0],
                              linestyle="none",
                              c='green',
                              marker='o')
    ax.set_xlabel('m',fontsize=fontsize_lg)
    ax.set_ylabel('k',fontsize=fontsize_lg)
    ax.set_title(r'$h({})$'.format(hgrid_id))

    ## for each h grid, take the 95% mass of m and k as the maximum of the m and k axis

    marginal_mk = joint_distr[:,:,hgrid_id]
    marginal_m = marginal_mk.sum(axis=0)
    marginal_k = marginal_mk.sum(axis=1)
    mmax = mgrid[(np.abs(marginal_m.cumsum()-mass_pct*marginal_m.cumsum().max())).argmin()]
    kmax = kgrid[(np.abs(marginal_k.cumsum()-mass_pct*marginal_k.cumsum().max())).argmin()]
    #ax.set_xlim([mmin,mmax])
    ax.set_ylim([kmax,kmin])
    plt.gca().invert_xaxis()
    #plt.gca().invert_yaxis()
    ax.view_init(30, 60)
    ax.legend([fake2Dline],
              ['Marginal cdf of the copula'],
              loc=0)

# %%

# %% [markdown]
# ## More to do:
#
# 1. Figure out median value of h and normalize c, m, and k by it

# %% [markdown]
# Given the assumption that the copula remains the same after aggregate risk is introduced, we can use the same copula and the marginal distributions to recover the full joint-distribution of the states.

# %% [markdown]
# ### Summary: what do we achieve after the transformation?
#
# - Using the DCT, the dimension of the policy and value functions are reduced from 3600 to 154 and 94, respectively.
# - By marginalizing the joint distribution with the fixed copula assumption, the marginal distribution is of dimension 64 compared to its joint distribution of a dimension of 3600.
#
#
#


#!/usr/bin/env python

import collections
import json
import os
import socket
import subprocess
import sys
import time

try:
    # Python 3
    from urllib.request import urlopen, URLError, HTTPError
except ImportError:
    # Python 2
    from urllib2 import urlopen, URLError, HTTPError

SanityCheckResult = collections.namedtuple(
    'SanityCheckResult', ['is_ok', 'message']
)

# (string, string)
Mount = collections.namedtuple("Mount", ['mount', 'device'])

DEFAULT_PORT_RETRIES = 2
DEFAULT_PORT_RETRY_DELAY = 5
DEFAULT_HTTP_RETRIES = 3
DEFAULT_HTTP_RETRY_DELAY = 1

def is_port_open(host, port):
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.settimeout(.1)
        s.connect((host, port))
        s.shutdown(socket.SHUT_RDWR)
        return True
    except socket.error:
        return False

def check_port(host, port, retries=DEFAULT_PORT_RETRIES, retry_delay=DEFAULT_PORT_RETRY_DELAY):
    if is_port_open(host, port):
        return True
    retry_number = 1
    while retry_number <= retries:
        time.sleep(retry_delay)
        if is_port_open(host, port):
            return True
        retry_number += 1
    return False

def run_command(command):
    try:
        output = subprocess.check_output(command, shell=True, stderr=subprocess.STDOUT)
        return True, output
    except subprocess.CalledProcessError as error:
        return False, error.output

def get_free_percentage_on_mount(mount):
    stats = os.statvfs(mount)
    return stats.f_bavail / float(stats.f_blocks) * 100

def test_free_space_on_mount(mount, free_percentage_required):
    free_percentage = get_free_percentage_on_mount(mount)
    if free_percentage_required > free_percentage:
        message = "Not enough free space on {mount}. Free space is {percentage:.1f}%".format(
            mount=mount,
            percentage=free_percentage
        )
        return SanityCheckResult(False, message)
    message = "Enough free space {mount}. Free space is {percentage:.1f}%".format(
        mount=mount,
        percentage=free_percentage
    )
    return SanityCheckResult(True, message)

def send_message(recipient, subject, body):
    """
    While sending mail is not reliable, mail delivery is async, and mail always
    exits successfully.
    """
    if isinstance(body, str):
        body = body.encode()
    process = subprocess.Popen(['mail', '-s', subject, recipient], stdin=subprocess.PIPE)
    process.communicate(body)

def get_dev_mounts():
    """
    Return a list of mounted drives.
    Uses heuristics to only return mounts that we want to check for free space.
    """

    results = []

    with open("/proc/mounts") as f:
        for line in f:
            if not line.strip():
                continue

            parts = line.split()
            mount = Mount(parts[1], parts[0])

            # /dev/loopX devices are mounted images that are expected to be 100% full,
            # so we don't want to monitor those. Snapd uses these extensively.
            if mount.device.startswith("/dev/loop"):
                continue

            # Standard /dev/ devices.
            elif mount.device.startswith("/dev"):
                results.append(mount)

            # Special cases, but also used by the system. There was a
            # problem with a device named `mysql` in some SoYouStart instances, so this is an extra check to also
            # find some devices with special names but mounted in a folder used by the system (`/var/` in this case)
            # Skip /var/lib/lxcfs, which can exist as a special fuse mount used for lxc.
            elif mount.mount.startswith("/var/") and mount.mount != "/var/lib/lxcfs":
                results.append(mount)

    return results

def test_global_free_percentage(default_free_space_percentage):
    for mount in get_dev_mounts():
        yield test_free_space_on_mount(mount.mount, default_free_space_percentage)

def test_ports(ports):
    response = []
    for port_config in ports:
        host, port = port_config['host'], int(port_config['port'])
        message = port_config['message']
        retries = int(port_config.get('retries', DEFAULT_PORT_RETRIES))
        retry_delay = float(port_config.get('retry_delay', DEFAULT_PORT_RETRY_DELAY))
        if not check_port(host, port, retries=retries, retry_delay=retry_delay):
            response.append(SanityCheckResult(False, message))
        else:
            message = "Port {}:{} is getting connections".format(host, port)
            response.append(SanityCheckResult(True, message))
    return response

def test_commands(commands):
    response = []
    for command in commands:
        command_passed, command_output = run_command(command['command'])
        if command_passed:
            command_message = "Command {} returned exit status 0".format(command['command'])
        else:
            command_message = command['message']
        response.append(SanityCheckResult(command_passed, '{}\n{}'.format(command_message, command_output)))
    return response

def report_is_ok(report):
    return all(check.is_ok for check in report)

def format_report(report):
    lines = []
    report = sorted(report, key= lambda check: check.is_ok)
    for check in report:
        status = "OK" if check.is_ok else "ERROR"
        lines.append("#. {status} {message}".format(status=status, message=check.message))
    return "\n".join(lines)

def sanity_check(json_data):
    report = []
    report.extend(
        test_global_free_percentage(float(json_data['free_percentage']))
    )
    report.extend(
        test_ports(json_data['ports'])
    )
    report.extend(
        test_commands(json_data['commands'])
    )

    if report_is_ok(report):
        if json_data.get('snitch'):
            report.extend(
                ping_http_endpoint(json_data['snitch'])
            )

    if not report_is_ok(report):
        report_text = format_report(report)
        send_message(json_data['send_report_to'], json_data['subject'], report_text)

    with open(json_data['report_file'], 'w') as f:
        report_text = format_report(report)
        f.write(report_text)

def ping_http_endpoint(url, max_retries=DEFAULT_HTTP_RETRIES, delay=DEFAULT_HTTP_RETRY_DELAY):
    report = []
    for count in range(1, max_retries + 1):
        try:
            response = urlopen(url, timeout=5)
        except (URLError, HTTPError) as e:
            report.append(SanityCheckResult(True, "Attempt {} to contact DMS failed due to:"
                                                  "\n{}".format(count, str(e))))
        else:
            if 200 <= response.getcode() < 300:
                report.append(SanityCheckResult(True, "Attempt {} to contact DMS succeeded.".format(count)))
                break
            report.append(SanityCheckResult(True, "Attempt {} to contact DMS returned code "
                                                  "{}".format(count, response.getcode())))
        time.sleep(delay)
    else:
        report.append(SanityCheckResult(False, "Couldn't send snitch after {} attempts".format(max_retries)))
    return report

if __name__ == "__main__":
    data_file = sys.argv[1]
    with open(data_file, 'r') as f:
        json_data = json.load(f)
    sanity_check(json_data)


import cv2
from io import BytesIO
#import pytesseract
from dateutil.parser import parse
from datetime import date
from pathlib import Path
import threading
from uploader.models import Receipt, ProcessedState
from django.core.files.base import ContentFile
# pytesseract.pytesseract.tesseract_cmd = "D:\\py-tesseract\\tesseract.exe"
from PIL import Image
import json
from django.core.files.uploadedfile import InMemoryUploadedFile
import os
import numpy as np
import urllib
import uuid
from django.core.files import File
from uploader.constants import VALID_EXT
import requests
from django.conf import settings
from uploader.constants import SET_FIELDS_VENDOR, SET_FIELDS_SUBTOTAL, SET_FIELDS_TOTAL, SET_FIELDS_DATE

def pill(im):
    buffer = BytesIO()
    im = im.convert('RGB')
    im.save(fp=buffer, format='JPEG')
    buff_val = buffer.getvalue()
    return ContentFile(buff_val)

def makeThumbnail(path):
    fileName = str(uuid.uuid4()) + "_thumbnail.jpg"
    MAX_SIZE = (100, 100)
    image = Image.open(path)

    image.thumbnail(MAX_SIZE)
    image.save(fileName)
    return fileName

def removeIfExist(fpath):
    if os.path.exists(fpath):
        os.remove(fpath)

def get_float(element: any) -> float:
    if element is None:
        return 0
    try:
        return float(element)
    except ValueError:
        return 0

def updateReceipt(receipt_obj, receipt_dict):
    set_fields = json.loads(receipt_obj.set_fields)
    # SET_FIELDS_VENDOR, SET_FIELDS_SUBTOTAL, SET_FIELDS_TOTAL, SET_FIELDS_DATE, SET_FIELDS_DESCRIPTION

    if "sum" in receipt_dict and not SET_FIELDS_TOTAL in set_fields:

        receipt_obj.total_amount = get_float(receipt_dict.get("sum", 0))
    elif SET_FIELDS_TOTAL in set_fields:
        receipt_obj.total_amount = get_float(set_fields.get(SET_FIELDS_TOTAL, 0))
    else:
        receipt_obj.total_amount = 0

    if "company" in receipt_dict and not SET_FIELDS_VENDOR in set_fields:
        receipt_obj.vendor = receipt_dict.get("company", '')
        if receipt_obj.category == 1:
            receipt_obj.category = receipt_dict.get("category", 1)
    elif SET_FIELDS_VENDOR in set_fields:
        receipt_obj.vendor = set_fields.get(SET_FIELDS_VENDOR, '')
    else:
        receipt_obj.vendor = ''

    if "subTotal" in receipt_dict and not SET_FIELDS_SUBTOTAL in set_fields:
        receipt_obj.sub_amount = get_float(receipt_dict.get("subtotal", 0))
    elif SET_FIELDS_SUBTOTAL in set_fields:
        receipt_obj.sub_amount = get_float(set_fields.get(SET_FIELDS_SUBTOTAL, 0))
    else:
        receipt_obj.sub_amount = 0

    if "date" in receipt_dict and not SET_FIELDS_DATE in set_fields:
        receipt_obj.receipt_date = receipt_dict.get(
            "date", date.today()) or date.today()
        receipt_obj.receipt_date_datetime = receipt_dict.get(
            "date", date.today()) or date.today()

    elif SET_FIELDS_DATE in set_fields:
        receipt_obj.receipt_date = set_fields.get(
            SET_FIELDS_DATE, date.today()) or date.today()
        receipt_obj.receipt_date_datetime = parse(
            set_fields.get(SET_FIELDS_DATE, str(date.today()))) or str(date.today())
    else:
        receipt_obj.receipt_date = date.today()
        receipt_obj.receipt_date_datetime = date.today()

    receipt_obj.status = ProcessedState.POPULATED
    receipt_obj.save()

class ReceiptProcessorThread(threading.Thread):
    def __init__(self, path=None, pk=None):
        super().__init__()
        self.path = path
        self.pk = pk

    def run(self):
        self.readImage(self.path)

    def url_to_image(self, url):
        if not url:
            return None

        if url[0] == "/":
            return url

        resp = urllib.request.urlopen(url)
        image = np.asarray(bytearray(resp.read()), dtype="uint8")
        image = cv2.imdecode(image, cv2.IMREAD_COLOR)
        fileName = str(uuid.uuid4()) + ".jpg"

        cv2.imwrite(fileName, image)

        return fileName

    def readImage(self, path):
        if settings.DONT_READ_RECEIPT:
            return
        receipt_obj = Receipt.objects.get(pk=self.pk)

        if not receipt_obj.file.name:
            return

        localFile = self.url_to_image(receipt_obj.base_image_url)
        f = Path(localFile)
        fName = f.stem
        fExt = f.suffix

        if not fExt in VALID_EXT:
            receipt_obj.status = ProcessedState.PARSING_DONE
            receipt_obj.save()
            return

        croppedName = "{fName}-cropped{fExt}".format(fExt=fExt, fName=fName)

        try:
            im = localFile
            im = Image.open(path)
            crop = json.loads(receipt_obj.crop)
            im1 = im.rotate(int(crop.get("rotation", 0)) * -90, expand=1)
            if crop.get("x", None) and crop.get("y", None) and crop.get("width", None) and crop.get("height", None):
                im1 = im1.crop((crop["x"], crop["y"], crop["x"] +
                               crop["width"], crop["y"] + crop["height"]))

            receipt_obj.cropped_file.save(croppedName,
                                          InMemoryUploadedFile(
                                              pill(im1), None, croppedName, 'image/jpeg', im1.tell, None).file
                                          )
            receipt_obj.save()
        except Exception as e:
            raise e
        finally:
            removeIfExist(localFile)

        # Download because old image is in memory only
        localFile = self.url_to_image(receipt_obj.cropped_image_url)

        thumb = makeThumbnail(localFile)
        thumb_file = open(thumb, 'rb')
        receipt_obj.thumbnail_file.save(thumb, File(thumb_file))
        thumb_file.close()
        removeIfExist(thumb)

        try:
            with open(localFile, 'rb') as f:
                files = {'file': ('upload.jpeg', open(localFile, 'rb'), 'image/jpeg'),
                         'Content-Disposition': 'form-data; name="file"; filename="' + 'upload.jpeg' + '"',
                         'Content-Type': 'multipart/form-data'}
                r = requests.post(settings.PARSE_URL, files=files)
                if (r.status_code == 200):
                    updateReceipt(receipt_obj, r.json())
        except Exception as e:
            receipt_obj.status = ProcessedState.PARSING_DONE
            receipt_obj.save()
            raise e
        finally:
            removeIfExist(localFile)
            removeIfExist(croppedName)


from collections import defaultdict

ANY_USER = 'ANY_USER'
ADMIN_GROUP = 'admin_users'
GROUP_PREFIX = '@'

class Authorizer:
    def __init__(self, app_allowed_users, admin_users, groups_provider):
        if ANY_USER in app_allowed_users:
            self._app_auth_check = AnyUserAuthorizationCheck()
        else:
            self._app_auth_check = ListBasedAuthorizationCheck(app_allowed_users)

        self._admin_users = admin_users
        self._groups_provider = groups_provider

    def is_allowed_in_app(self, user_id):
        return self._app_auth_check.is_allowed(user_id)

    def is_admin(self, user_id):
        return user_id in self._admin_users

    def is_allowed(self, user_id, allowed_users):
        if not allowed_users:
            return False

        if (allowed_users == ANY_USER) or (ANY_USER in allowed_users):
            return True

        if user_id in allowed_users:
            return True

        user_groups = self._groups_provider.get_groups(user_id)
        if not user_groups:
            return False

        for group in user_groups:
            if (GROUP_PREFIX + group) in allowed_users:
                return True

        return False

class ListBasedAuthorizationCheck:
    def __init__(self, allowed_users) -> None:
        self.allowed_users = set(allowed_users)

    def is_allowed(self, user_id):
        return user_id in self.allowed_users

class AnyUserAuthorizationCheck:
    @staticmethod
    def is_allowed(user_id):
        return True

class EmptyGroupProvider:

    def get_groups(self, user):
        return []

def _flatten_groups(groups):
    result = {}

    for group in groups.keys():
        group_users = set()
        visited_groups = set()

        queue = [group]

        while queue:
            current_group = queue.pop()
            if current_group in visited_groups:
                continue
            visited_groups.add(current_group)

            if current_group in result:
                group_users.update(result[current_group])
                continue

            current_group_members = groups[current_group]
            for member in current_group_members:
                if member.startswith(GROUP_PREFIX):
                    queue.append(member[1:])
                else:
                    group_users.add(member)

        result[group] = group_users

    return result

class PreconfiguredGroupProvider:

    def __init__(self, groups) -> None:
        self._user_groups = defaultdict(list)

        flat_groups = _flatten_groups(groups)
        for group, users in flat_groups.items():
            for user in users:
                self._user_groups[user].append(group)

    def get_groups(self, user):
        return self._user_groups[user]

class CombinedGroupProvider:

    def __init__(self, *other_providers) -> None:
        self._other_providers = list(other_providers)

    def get_groups(self, user):
        groups = set()
        for provider in self._other_providers:
            provider_groups = provider.get_groups(user)
            if provider_groups:
                groups.update(provider_groups)

        return groups

def create_group_provider(user_groups, authenticator, admin_users):
    if admin_users:
        if user_groups is None:
            user_groups = {ADMIN_GROUP: admin_users}
        elif ADMIN_GROUP not in user_groups:
            user_groups[ADMIN_GROUP] = admin_users

    if not user_groups:
        if authenticator:
            return authenticator
        return EmptyGroupProvider()

    preconfigured_groups_provider = PreconfiguredGroupProvider(user_groups)
    if not authenticator:
        return preconfigured_groups_provider

    return CombinedGroupProvider(preconfigured_groups_provider, authenticator)


import numpy as np
import pylab as plt
import time
import random
from scipy.sparse.linalg import eigsh
from scipy.sparse.linalg import LinearOperator
np.set_printoptions(precision=5)
from scipy.optimize import curve_fit

f=1
g=1

class MPS:
    def __init__(self,size,system_d,chi):

        '''
        -size is just lenght of chain
        -system_d is the single site Hilbert space dimension
        -chi is bond dimension limit

        '''

        self.size=size
        self.system_d=system_d
        self.chi=chi

    def generate_all_up_or_down(self,direction):
        A=[0]*self.size
        self.A=[0]*self.size
        for i in range(self.size):
            A[i]=np.zeros([1,self.system_d,1])
            if direction=="up":
                A[i][0,0,0]=1
            elif direction=="down":
                A[i][0,1,0]=1
        self.A[:]=A[:]
        self.sh=[0]*self.size

    def evaluate_operator(self,MPO1):
        L=self.size
        for i in range(L):
            if i==0:
                l1=np.tensordot(self.A[i],np.tensordot(MPO1.O[i],self.A[i].conj(),axes=([1],[1])),axes=([1],[0]))
            else:
                l1=np.tensordot(l1,np.tensordot(self.A[i],np.tensordot(MPO1.O[i],self.A[i].conj(),axes=([1],[1])),axes=([1],[0])),axes=([1,3],[0,2]))
                l1=np.transpose(l1,(0,2,1,3))
        return l1

    def generate_small(self,cnew):

        '''
        Does successive SVDS to write the wavefunction in right canonical form

        Returns
        -----------

        -list A which contains the matrices. for right canonical form, A[0] is the first matrix.
        -List sh which contains the diagonal matrices s.
        '''

        l1=self.size
        d=self.system_d
        sh=[0]*l1
        n=d**(l1)
        m=1
        A=[0]*l1
        self.A_initial=[0]*l1
        self.A=[0]*l1
        for i in range(l1):
            phi=np.zeros([int(n/d),m*d],complex)
            phi[:,:]=np.reshape(cnew,[int(n/d),m*d])[:,:]

            # phi=np.zeros([2**(i+1),2**(l-(i+1))],complex)
            # phi[:,:]=np.reshape(cnew,[2**(i+1),2**(l-(i+1))],'F')[:,:]

            u1,s1,v1=np.linalg.svd(phi,full_matrices=False)
            n=np.size(u1,axis=0)
            m=np.size(u1,axis=1)

            sh[l1-1-i]=s1

            cnew=np.zeros([n,m],complex)
            cnew[:,:]=np.matmul(u1,np.identity(m)*sh[l1-1-i])[:,:]
            cnew.reshape(-1)
            if i<l1/2:
                b=np.reshape(v1,[d**(i+1),d,d**i])
            else:
                b=np.reshape(v1,[d**(l-i-1),d,d**(l-i)])
            A[l1-1-i]=b

        self.A[:]=A[:]
        self.sh=sh
        self.A_initial[:]=A[:]
        self.sh_initial=sh
        self.canonical="right"

    def check_canonical_right(self):
        test_number=0
        for i in range(self.size):
            test_matrix=np.tensordot(self.A[i],self.A[i].conj(),axes=([1,2],[1,2]))
            if (abs(abs(np.eye(np.size(self.A[i],axis=0)))-abs(test_matrix))>10**-8).all():
                test_number+=1
        if test_number==0:
            print("A is right canonical")
        else:
            print("A is not right canonical")
            print(test_number)

    def check_canonical_left(self):
        test_number=0
        for i in range(self.size):
            test_matrix=np.tensordot(self.A[i],self.A[i].conj(),axes=([0,1],[0,1]))
            if (abs(abs(np.eye(np.size(self.A[i],axis=2)))-abs(test_matrix))>10**-8).all():
                test_number+=1
        if test_number==0:
            print("A is left canonical")
        else:
            print("A is not right canonical")
            print(test_number)

    def combine_pair(self,l):
        '''
        -l is the position of the first site comprising the pair
        '''
        theta=np.tensordot(self.A[l],self.A[l+1],axes=([2],[0]))
        self.theta=theta

    def generate_R(self,MPO):
        '''
        -MPO is the MPO of the hamiltonian

        generates a list of R's on all sites

        the first 2 indices of the tensor represent the upper left/right indices on the graph.
        indices 3 and 4 represent the MPO left/right indices so that they are always 3x3.
        etc.
        '''
        R=[0]*self.size
        for i in range(self.size):
            R_bottom=np.tensordot(MPO.W[i],self.A[i].conj(),axes=([3],[1]))
            R[i]=np.tensordot(self.A[i],R_bottom,axes=([1],[2]))
        self.R=R

    def generate_R_environment(self,l):
        '''

        Parameters
        ----------
        l : position at which to evaluate the environment

        Returns
        -------
        the right environment tensor R saved as an antribute R_environment

        '''
        if l==self.size-1:
            r=self.R[-1]
        else:
            for i in range(l,self.size-1):
                if i==l:
                    r=np.tensordot(self.R[i],self.R[i+1],axes=([1,3,5],[0,2,4]))
                    r=np.transpose(r,(0,3,1,4,2,5))
                else:
                    r=np.tensordot(r,self.R[i+1],axes=([1,3,5],[0,2,4]))
                    r=np.transpose(r,(0,3,1,4,2,5))
        self.R_environment=r

    def generate_L_environment(self,l):
        '''

        Parameters
        ----------
        l : position at which to evaluate the environment

        Returns
        -------
        the right environment tensor R saved as an antribute R_environment

        '''
        if l==0:
            L=self.R[0]
        else:
            for i in range(l):
                if i==0:
                    L=np.tensordot(self.R[i],self.R[i+1],axes=([1,3,5],[0,2,4]))
                    L=np.transpose(L,(0,3,1,4,2,5))
                else:
                    L=np.tensordot(L,self.R[i+1],axes=([1,3,5],[0,2,4]))
                    L=np.transpose(L,(0,3,1,4,2,5))
        self.L_environment=L

    def generate_and_solve_Heff(self,l,MPO1):
        '''

        Parameters
        ----------
        l : position of the first site comprising the pair.

        MPO1: MPO of the Hamiltonian

        Returns
        -------
        Heff saved as an attribute Heff

        '''
        if l==0:
            self.generate_R_environment(l+2)
            Heff=np.tensordot(MPO1.W[l],MPO1.W[l+1],axes=([1],[0]))
            Heff=np.transpose(Heff,(0,3,1,2,4,5))
            Heff=np.tensordot(Heff,self.R_environment,axes=([1],[2]))
            Heff=np.transpose(Heff,(0,1,3,5,2,4,6,7,8,9))
            self.Heff=Heff
            self.Heff=np.reshape(Heff,[self.system_d**2*self.A[l].shape[0]*self.A[l+2].shape[0],self.system_d**2*self.A[l].shape[0]*self.A[l+2].shape[0]])
            self.e,self.v=eigsh(self.Heff,k=2,which='SA')
            print(self.e)
            self.Q=self.v[:,0].reshape([self.A[l].shape[0],self.system_d,self.system_d,self.A[l+2].shape[0]])

        elif (l==self.size-2):
            self.generate_L_environment(l-1)
            Heff=np.tensordot(MPO1.W[l],MPO1.W[l+1],axes=([1],[0]))
            Heff=np.transpose(Heff,(0,3,1,2,4,5))
            Heff=np.tensordot(self.L_environment,Heff,axes=([3],[0]))
            Heff=np.transpose(Heff,(0,1,2,6,8,5,3,4,7,9))
            self.Heff=Heff
            self.Heff=np.reshape(Heff,[self.system_d**2*self.A[l-1].shape[2]*self.A[l+1].shape[2],self.system_d**2*self.A[l-1].shape[2]*self.A[l+1].shape[2]])
            self.e,self.v=eigsh(self.Heff,k=2,which='SA')
            print(self.e)
            self.Q=self.v[:,0].reshape([self.A[l-1].shape[2],self.system_d,self.system_d,self.A[l+1].shape[2]])
        else:
            self.generate_R_environment(l+2)
            self.generate_L_environment(l-1)
            Heff=np.tensordot(MPO1.W[l],MPO1.W[l+1],axes=([1],[0]))
            Heff=np.transpose(Heff,(0,3,1,2,4,5))
            Heff=np.tensordot(Heff,self.R_environment,axes=([1],[2]))
            Heff=np.tensordot(self.L_environment,Heff,axes=([3],[0]))
            Heff=np.transpose(Heff,(0,1,2,5,7,9,10,11,3,4,6,8,12,13))

            self.Heff=Heff
            self.Heff=np.reshape(Heff,[self.system_d**2*self.A[l-1].shape[2]*self.A[l+2].shape[0],self.system_d**2*self.A[l-1].shape[2]*self.A[l+2].shape[0]])
            self.e,self.v=eigsh(self.Heff,k=2,which='SA')
            print(self.e)
            self.Q=self.v[:,0].reshape([self.A[l-1].shape[2],self.system_d,self.system_d,self.A[l+2].shape[0]])

    def left_svd_Q_and_update_R(self,l,MPO1):
        '''

        Parameters
        ----------
        l : position of the first site comprising the pair.
        MPO1 : the hamiltonian in MPO form.

        Returns
        -------
        None.

        '''
        test1=self.Q.reshape(-1)

        test2=np.reshape(test1,[self.Q.shape[0]*self.Q.shape[1],self.Q.shape[2]*self.Q.shape[3]])
        u,s,v=np.linalg.svd(test2,full_matrices=False)
        u=u[:,0:min(self.chi,u.shape[1])]
        self.A[l]=u.reshape([self.Q.shape[0],self.system_d,int(len(u.reshape(-1))/(self.Q.shape[0]*self.system_d))])
        self.sh[l]=s[0:min(self.chi,len((s)))]

        R_bottom=np.tensordot(MPO1.W[l],self.A[l].conj(),axes=([3],[1]))
        self.R[l]=np.tensordot(self.A[l],R_bottom,axes=([1],[2]))

    def right_svd_Q_and_update_R(self,l,MPO1):
        test1=self.Q.reshape(-1)
        test2=np.reshape(test1,[self.Q.shape[0]*self.Q.shape[1],self.Q.shape[2]*self.Q.shape[3]])
        u,s,v=np.linalg.svd(test2,full_matrices=False)
        v=v[0:min(self.chi,v.shape[0]),:]
        self.A[l+1]=v.reshape([int(len(v.reshape(-1))/(self.system_d*self.Q.shape[3])),self.system_d,self.Q.shape[3]])
        self.sh[l]=s[0:min(self.chi,len((s)))]
        R_bottom=np.tensordot(MPO1.W[l+1],self.A[l+1].conj(),axes=([3],[1]))
        self.R[l+1]=np.tensordot(self.A[l+1],R_bottom,axes=([1],[2]))

class MPO:
    def __init__(self,size,system_d):

        '''
        -size is just lenght of chain
        -system_d is the single site Hilbert space dimension

        '''
        self.size=size
        self.system_d=system_d

    def generate(self,O,positions):

        '''
        -O is a list of operators acting on sites
        -positions is a list of site positions. It should correspond with O
         i.e. O[0] acts on the site positions[0]. 0 is first site.
        '''
        MPO=[np.eye(self.system_d)]*self.size
        for i in range(len(positions)):
            MPO[positions[i]]=O[i]
        self.O=MPO
        self.positions=positions

    def generate_hamiltonian_transverse_ising(self):
        '''
        Generates a list of MPO's representing the transverse field ising model hamiltonian.
        The indices of each MPO are as [left MPO index, right MPO index, up physical index, down physical index]
        The first element is a row vector and the last a column vector.
        '''
        SzO=np.array([[1,0],[0,-1]])
        SxO=np.array([[0,1],[1,0]])
        Wlist=[]
        d=self.system_d

        W1=np.zeros([1,3,d,d])
        W1[0,0]=np.eye(d)
        W1[0,1]=-SzO
        W1[0,2]=-g*SxO
        Wlist.append(W1)

        WL=np.zeros([3,1,d,d])
        WL[0,0]=-g*SxO
        WL[1,0]=SzO
        WL[2,0]=np.eye(d)

        W=np.zeros([3,3,d,d])
        W[0,:,:,:]=W1[0,:,:,:]
        W[:,-1,:,:]=WL[:,0,:,:]

        for i in range(self.size-2):
            Wlist.append(W)

        Wlist.append(WL)

        self.W=Wlist

l=64

wf=MPS(l,2,chi=10)
wf.generate_all_up_or_down("down")

SzO=np.eye(2)
SzO[1,1]=-1
SxO=np.zeros([2,2])
SxO[0,1]=1
SxO[1,0]=1
MPO1=MPO(l,2)
MPO1.generate_hamiltonian_transverse_ising()
# MPO1.generate([SzO],[4])

#test=wf.evaluate_operator(MPO1)

wf.generate_R(MPO1)

for j in range(3):
    for i in range(l-1):
        wf.generate_and_solve_Heff(i,MPO1)
        wf.left_svd_Q_and_update_R(i,MPO1)

    for i in range(l-1):
        wf.generate_and_solve_Heff(l-2-i,MPO1)
        wf.right_svd_Q_and_update_R(l-2-i,MPO1)
S=np.zeros(l-1)
for i in range(l-1):
    wf.generate_and_solve_Heff(i,MPO1)
    wf.left_svd_Q_and_update_R(i,MPO1)
    S[i]=-sum(wf.sh[i][:]**2*np.log(wf.sh[i][:]**2))

def f(x,c,g):
    return (c/6)*np.log((l/np.pi)*np.sin(np.pi*x/l))+2*g

xplot=np.arange(1,l)

param=curve_fit(f,xplot,S)
plt.plot(xplot,S,'*')
plt.plot(xplot,f(xplot,param[0][0],param[0][1]))


from pathlib import Path

from itaxotools.common.utility import AttrDict

from ..types import Type, SequenceReader, ColumnFilter, FileFormat
from .common import Object, Property, Item
from .input_file import InputFileModel

class SequenceModel(Object):
    path = Property(Path)
    reader = Property(SequenceReader)

    def __init__(self, path, reader=SequenceReader.TabfileReader):
        super().__init__()
        self.path = path
        self.name = path.name
        self.reader = reader

    def __repr__(self):
        return f'{".".join(self._get_name_chain())}({repr(self.name)})'

class SequenceModel2(Object):
    file_item = Property(Item, None)

    def __init__(self, file_item=None):
        super().__init__()
        self.file_item = file_item
        if file_item:
            self.name = f'Sequences from {file_item.object.path.name}'

    def __repr__(self):
        return f'{".".join(self._get_name_chain())}({repr(self.name)})'

class Fasta(SequenceModel2):
    has_subsets = Property(bool, False)
    parse_organism = Property(bool, False)

    def __init__(self, file_item, parse_organism=False):
        assert isinstance(file_item.object, InputFileModel.Fasta)
        super().__init__(file_item)
        info = file_item.object.info
        self.has_subsets = info.has_subsets
        self.parse_organism = parse_organism

    def as_dict(self):
        return AttrDict(
            type = FileFormat.Fasta,
            path = self.file_item.object.path,
            parse_organism = self.parse_organism,
        )

class Tabfile(SequenceModel2):
    index_column = Property(int, -1)
    sequence_column = Property(int, -1)
    index_filter = Property(ColumnFilter, ColumnFilter.All)
    sequence_filter = Property(ColumnFilter, ColumnFilter.All)

    def __init__(self, file_item):
        assert isinstance(file_item.object, InputFileModel.Tabfile)
        super().__init__(file_item)
        info = file_item.object.info
        self.index_column = self._header_get(info.headers, info.individuals)
        self.sequence_column = self._header_get(info.headers, info.sequences)

    @staticmethod
    def _header_get(headers: list[str], field: str):
        try:
            return headers.index(field)
        except ValueError:
            return -1

    def as_dict(self):
        return AttrDict(
            type = FileFormat.Tabfile,
            path = self.file_item.object.path,
            index_column = self.index_column,
            sequence_column = self.sequence_column,
        )


"""
USAGE: fodt_clean.py  path/to/file.fodt
"""

import sys
import lxml.etree as et
from pathlib import Path

if len(sys.argv) != 2:
    raise SystemExit(__doc__)

filename = Path(sys.argv[1])
tree = et.fromstring(filename.read_bytes())

for binary_data in tree.xpath("//office:binary-data", namespaces=tree.nsmap):
    draw_image = binary_data.getparent()
    draw_image.getparent().remove(draw_image)

xml_str = et.tostring(tree, pretty_print=True, encoding="unicode")
with filename.open("wt") as fp:
    fp.write('<?xml version="1.0" encoding="UTF-8"?>\n\n')
    fp.write(xml_str)


import unittest
import pymongo
import sys
sys.path.append('../../')
sys.path.append('../')

from app.models.urlshortener import urlShortener

class TestUrlShortener(unittest.TestCase):
    """ test the model of url handler """
    def setUp(self):
        connection = pymongo.MongoClient('mongodb://localhost:27017/')
        self.database = connection.test
        self.collection = self.database.urlshortener
        self.collection.drop()
        self.urlShortener = urlShortener(self.collection)

    def test_saveUrl_Unique(self):
        # setup
        longurl = "http://www.google.com"
        short_url = "gl"
        author = 'author'
        result, reason = self.urlShortener.saveUrl(short_url, longurl, author)

        # Assertions
        self.assertEqual(result, True)
        doc = self.collection.find_one({'_id': short_url})
        self.assertEqual(doc['_id'], short_url)
        self.assertEqual(doc['longurl'], longurl)
        self.assertEqual(doc['author'], author)
        self.assertEqual(doc['clicks'], 0)

        # cleanup so that next time we dont get duplicateKeyError
        self.collection.delete_one({'_id': short_url})

    def test_saveUrl_duplicate(self):
        shortUrl = 'orig'
        url = 'http://www.google.com'
        urldup = 'http://www.yahoo.com'
        author = 'author'

        self.urlShortener.saveUrl(shortUrl, url, author)
        result, reason = self.urlShortener.saveUrl(shortUrl, urldup, author)

        self.assertEqual(result, False)
        self.assertEqual(reason, 'DuplicateKeyError')
        doc = self.collection.find_one({'_id': shortUrl})
        self.assertEqual(doc['_id'], shortUrl)
        self.assertEqual(doc['longurl'],  url)
        self.assertEqual(doc['clicks'], 0)

        self.collection.delete_one({'_id': shortUrl})

    def test_findUrl_Existing(self):
        shortUrl = 'findUrl'
        url = 'http://www.google.com'
        author = 'author'
        self.urlShortener.saveUrl(shortUrl, url, author)

        resultUrl = self.urlShortener.findUrl(shortUrl)

        self.assertEqual(resultUrl, url)

        self.collection.delete_one({'_id': shortUrl})

    def test_findUrl_NonExisting(self):
        shortUrl = 'findDuplicate'

        resultUrl = self.urlShortener.findUrl(shortUrl)

        self.assertEqual(resultUrl, None)

    def test_removeURL_Existing(self):
        shortUrl = 'removeURL'
        url = 'http://www.google.com'
        author = 'author'
        self.urlShortener.saveUrl(shortUrl, url, author)

        result = self.urlShortener.removeUrl(shortUrl)

        self.assertEqual(result, True)

    def test_removeURL_NonExisting(self):
        shortURL = 'NonExisting'

        result = self.urlShortener.removeUrl(shortURL)

        self.assertEqual(result, True)

    def test_increment_view_count(self):
        """  Check that view count is incremented by one each time"""
        shortUrl = 'increment_url'
        url = 'http://www.google.com'
        author = 'author'
        self.urlShortener.saveUrl(shortUrl, url, author)

        self.urlShortener.increment_visited_count(shortUrl)
        self.urlShortener.increment_visited_count(shortUrl)

        doc = self.urlShortener.get_doc_from_shorturl(shortUrl)
        self.assertEqual(int(doc['clicks']), 2)

        self.urlShortener.removeUrl(shortUrl)

    def test_find_urls_of_user(self):
        shortUrl = 'shorturl1'
        longurl = 'http://www.google.com'
        author = 'author'
        self.urlShortener.saveUrl(shortUrl, longurl, author)

        iterator = self.urlShortener.find_url_of_user(author, 1)

        for doc in iterator:
            self.assertEqual(doc['_id'], shortUrl)
            self.assertEqual(doc['longurl'], longurl)
            self.assertEqual(doc['author'], author)

        self.urlShortener.removeUrl(shortUrl)

    def test_generateRandom(self):
        """ test the random number generator """

        # commented as of now as its failing randomly. Race due to
        # monkey patching ???
        # self.assertEqual(len(self.urlShortener.generateShortUrl()), 6)
        # self.assertEqual(len(self.urlShortener.generateShortUrl(7)), 7)

        self.assertEqual(self.urlShortener.generateShortUrl().isalnum(), True)


#!/usr/bin/env python
#-*- coding: utf-8 -*-

#Dsimonet

#from __future__ import division

######################
"""	LEG SMOOTH """
######################

"""
Increasing Speed should augment velocity or reduce time beetween the start and the arrive of position
We have to keep trace of the last update. Like this mouvement will be calcultate over time/distance = speed and not just called where ever it want
"""

from leg import *
import time

class LegSmooth(Leg) :

	"""
	even if it seems to be  useless to use this class to pilote fast motor (ex with speed >= 450)
	with this class a small movement (short distance) will be synchronized with
	long movement (long distance) over the different parts of the leg.
	this could be useful for nice and graceful movement
	short story --> all parts of the leg will arrive at the goal position at the same time and
	synchronized no matter distance traveled. All synchronized on the maximum time obviously
	"""

	lastLegUpdated = 0

	def __init__(self, _phi, _a, _b):

		#init inherited class
		Leg.__init__(self, _phi, _a, _b)

		self.smoothing = True

		#values
		self.speed = float(250)#mm/s or degres/s --> 9G servo is 0.12second/ 60degree = 500 °/S
		self.duration = 0
		self.timeBegin = 0

		self.phiValue = 0.0
		self.aValue = 0.0
		self.bValue = 0.0

		self.phiFrom = 0.0
		self.aFrom = 0.0
		self.bFrom = 0.0

		self.phiTo = 0.0
		self.aTo = 0.0
		self.bTo = 0.0

		#correction value
		self.cIso = [0,0,0]
		self.cDelta = [1,1,1]

		#Constant
		self.position(0,0,0)

	#METHODES STATICS

	@staticmethod
	def updateAll():
		for leg in LegSmooth:
			leg.update()

	@staticmethod
	def updateNextLeg():
		legToUpdate = (LegSmooth.lastLegUpdated+1)%len(LegSmooth._registry)
		LegSmooth._registry[ legToUpdate ].update()
		LegSmooth.lastLegUpdated += 1

	@staticmethod
	def allReady():

		# i = [0, 0, 0, 0, 0, 0]
		# j = 0
		# for leg in LegSmooth :
		# 	i[j] = (int)(leg.ready())
		# 	j = j +1
		# print i

		for leg in LegSmooth:
			if not leg.ready() :
				return False

		return True

	@staticmethod
	def setAllSpeed(_v):
		while(not LegSmooth.allReady) :
			pass
		for leg in LegSmooth:
			leg.setSpeed(_v)

	@staticmethod
	def positionSync():
		"""
		To sync arrived time. We simply look for the longest duration of all leg and apply it to all leg
		"""
		distList = []

		for leg in LegSmooth:
			distList.append( leg.duration )
		for leg in LegSmooth :
			leg.duration = max(distList)

	@staticmethod
	def setSpeedAll(value):
		for leg in LegSmooth:
			leg.setSpeed(value)

	@staticmethod
	def allOff() :
		for leg in LegSmooth :
			leg.off()

	@staticmethod
	def waitUntil() :
		while not LegSmooth.allReady() :
			LegSmooth.updateAll()

	@staticmethod
	def smoothActivate(_v) :
		for leg in LegSmooth :
			leg.smoothing = _v

	#METHODES

	def setCorrectionValues(self, _c, _d) :
		self.cIso = _c
		self.cDelta = _d

	def setSpeed(self, value):
		"""
		no matter whater what you think, but float() cast avoid spicy bug
		"""
		if not value == 0:
			self.speed = float(value)

	def position(self, _phi, _a, _b):
		"""
		Set position of the leg by giving it 3 values
		"""

		#assign new goal value
		self.phiTo = _phi
		self.aTo = _a
		self.bTo = _b

		if self.smoothing :
			if self.ready() :
				#if we are ready we can go for a trip from where we are
				#so position and time are reseted and are our new starting point

				self.timeBegin = time.clock()

				self.phiFrom = self.phiValue
				self.aFrom = self.aValue
				self.bFrom = self.bValue

				#looking for the longest distance we have to travel on each 3 motor. It's our duration for all 3 motors
				# This is because all motor for each leg is sync.
				maxDuration = max( [abs(self.phiTo - self.phiFrom), abs(self.aTo - self.aFrom), abs(self.bTo - self.bFrom)] );
				if maxDuration < 0.01 :
					Leg.position(self, self.phiTo*self.cDelta[0]+self.cIso[0], self.aTo*self.cDelta[1]+self.cIso[1], self.bTo*self.cDelta[2]+self.cIso[2])
					self.duration = 0
				else:
					self.duration = float ( maxDuration ) / self.speed

			else:
				# else it's a bit triky.
				# we find the longest distance between *actual* position and to the new distance (it the trip we have to do)
				distList = [abs(self.phiTo - self.phiValue), abs(self.aTo - self.aValue), abs(self.bTo - self.bValue)]
				# and our new duration is the time we already done (cause we can't get it back), and duration between actual position and the new position
				duration = time.clock()-self.timeBegin + float ( max(distList) / self.speed )

				if duration < 0.01 :
					Leg.position(self, self.phiTo*self.cDelta[0]+self.cIso[0], self.aTo*self.cDelta[1]+self.cIso[1], self.bTo*self.cDelta[2]+self.cIso[2])
					self.duration = 0
				else:
					self.duration = duration

		else :
			Leg.position(self, self.phiTo*self.cDelta[0]+self.cIso[0], self.aTo*self.cDelta[1]+self.cIso[1], self.bTo*self.cDelta[2]+self.cIso[2])

	def update(self):
		"""
		compute position from last position trough a ease function
		ease(t, b, c, d)
		t is the current time (or position) of the tween.
		b is the beginning value of the property.
		c is the change between the beginning and destination value of the property.
		d is the total time of the tween.
		"""

		"""
		Bug : If travel duration is shorter than update time. Some leg never receive update because
		all leg are marked as ready befor travel were effectively done
		"""

		if ( time.clock()-self.timeBegin ) < self.duration :
			self.phiValue = ease.easeInOutQuad(time.clock()-self.timeBegin, self.phiFrom, self.phiTo-self.phiFrom, self.duration)
			self.aValue = ease.easeInOutQuad(time.clock()-self.timeBegin, self.aFrom, self.aTo-self.aFrom, self.duration)
			self.bValue = ease.easeInOutQuad(time.clock()-self.timeBegin, self.bFrom, self.bTo-self.bFrom, self.duration)
		else :
			self.phiValue = self.phiTo
			self.aValue = self.aTo
			self.bValue = self.bTo

		#Moving motor with inherited methode to the computed value
		Leg.position(self, self.phiValue*self.cDelta[0]+self.cIso[0], self.aValue*self.cDelta[1]+self.cIso[1], self.bValue*self.cDelta[2]+self.cIso[2])

	def ready(self):
		if (time.clock()-self.timeBegin) < self.duration :
			return False
		else:
			return True

	def isSoftLimiteActive(self, _v):
		pass

	def setLimits(self, _minPhi, _maxPhi, _minA, _maxA, _minB, _maxB) :
		pass

# excuted if this doc is not imported
# for testing purpose only

if __name__ == '__main__':

	import random

	leg_1 = LegSmooth(24,25,26)
	leg_2 = LegSmooth(20,21,22)
	leg_3 = LegSmooth(16,17,18)
	leg_4 = LegSmooth(8, 9, 10)
	leg_5 = LegSmooth(4, 5, 6)
	leg_6 = LegSmooth(0, 1, 2)

	LegSmooth.setSpeedAll(100)
	LegSmooth.smoothActivate(False)

	for leg in LegSmooth :
		leg.position(0,0,0)

	while not LegSmooth.allReady() :
		LegSmooth.updateAll()

	try :
		while True :
			if LegSmooth.allReady() :
				for leg in LegSmooth :
					leg.position(random.randint(-30,30),random.randint(-90,90),random.randint(-90,90))

				LegSmooth.waitUntil()
				time.sleep(0.5)

	except KeyboardInterrupt:

		LegSmooth.smoothActivate(True)
		LegSmooth.setSpeedAll(50)

		for leg in LegSmooth :
			leg.position(0,0,0)

		LegSmooth.waitUntil()

		time.sleep(0.1)

		LegSmooth.allOff()

		time.sleep(0.1)


from com.dtmilano.android.viewclient import ViewClient, AdbClient
import time

def get_view_client():
  device, serialno = ViewClient.connectToDeviceOrExit()
  vc = ViewClient(device, serialno)
  return vc

def maybe_click(vc, view_id):
  el = vc.findViewById(view_id)
  if el:
    el.touch()
    return True
  return False

def target():
  print("targeting wife ...")
  vc = get_view_client()
  if maybe_click(vc, 'co.hinge.app:id/like_button'):
    return True
  elif maybe_click(vc, 'co.hinge.app:id/likeButton'):
    return True
  elif maybe_click(vc, 'co.hinge.app:id/undo_button'):
    return True
  elif maybe_click(vc, 'co.hinge.app:id/dialog_primary_button'):
    return True
  elif swoop():
    return True
  return False

def swoop():
  print("swooping!")
  vc = get_view_client()
  el = vc.findViewById('co.hinge.app:id/send_like_button')
  if el:
    el.touch()
    print("payload delivered")
    return True
  return False

if __name__ == '__main__':
  while True:
    try:
      if target():
        time.sleep(2)
        swoop()
      time.sleep(3)
    except:
      print("error")


#from re import S
from typing import Text
from flask import Flask, request, render_template

app = Flask(__name__)

@app.route("/", methods=["GET", "POST"])
def index():
    sistema = str(request.form.get("sistema"))
    ip = str(request.form.get("ip"))
    qm = str(request.form.get("qm"))
    cpu = str(request.form.get("CPU"))
    select = []
    if request.method == "GET":
        return render_template("index.html")
    else:
        select.append(f"\n Sistema Operacional: {sistema}")
        select.append(f"\n Endereço IP: {ip}")
        select.append(f"\n Quantidade de memória:{qm}")
        select.append(f"\n Quantidade de CPU: {cpu}")
        info = open("info.txt","w")
        info.writelines(select)
        info.close()
    return Text(select)
if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5050, debug=True)


from django.core.mail import EmailMultiAlternatives
from django.dispatch import receiver
from django.template.loader import render_to_string
from django.urls import reverse

def email_txn(email,template):
    print('*********************** email send template sratrted ***********************' )
    """
    Handles password reset tokens
    When a token is created, an e-mail needs to be sent to the user
    :param sender: View Class that sent the signal
    :param instance: View Instance that sent the signal
    :param reset_password_token: Token Model Object
    :param args:
    :param kwargs:
    :return:
    """
    # send an e-mail to the user
    context = {
        'username': email.username,
        'email': email.email,
        'txn': email.txn

    }

    # render email text
    email_message = render_to_string(template, context)

    msg = EmailMultiAlternatives(
        # title:
        "Transaction Confirmation for {title}".format(title="Altcoin"),
        # message:
        email_message,
        # from:
        "admin@altcoin.com",
        # to:
        [email.email]
    )
    msg.attach_alternative(email_message, "text/html")
    msg.send()


import os
from transformers import (AutoModelForSeq2SeqLM,
                          AutoTokenizer,
                          DataCollatorForSeq2Seq,
                          TrainingArguments,
                          Trainer)
from datasets import load_from_disk
import torch
from textSummarizer.entity import ModelTrainerConfig

class ModelTrainer(object):
    def __init__(self, config: ModelTrainerConfig):
        self.config = config

    def train(self):
        device = "cuda" if torch.cuda.is_available() else "cpu"
        model_ckpt = self.config.model_ckpt
        tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
        model_pegasus = AutoModelForSeq2SeqLM.from_pretrained(model_ckpt).to(device)
        seq2seq_data_collator = DataCollatorForSeq2Seq(tokenizer, model=model_pegasus)

        # loading dataset
        dataset_samsum_pt = load_from_disk(self.config.data_path)

        # initialize training args
        training_args = TrainingArguments(self.config.root_dir, num_train_epochs=self.config.num_train_epochs,
                                          warmup_steps=self.config.warmup_steps,
                                          per_device_train_batch_size=self.config.per_device_train_batch_size,
                                          per_device_eval_batch_size=self.config.per_device_train_batch_size,
                                          weight_decay=self.config.weight_decay, logging_steps=self.config.logging_steps,
                                          evaluation_strategy=self.config.evaluation_strategy,
                                          eval_steps=self.config.eval_steps, save_steps=self.config.save_steps,
                                          gradient_accumulation_steps=self.config.gradient_accumulation_steps
                                          )
        # begin training
        trainer = Trainer(
            model=model_pegasus, args=training_args,
            tokenizer=tokenizer, data_collator=seq2seq_data_collator,
            train_dataset=dataset_samsum_pt["train"],
            eval_dataset=dataset_samsum_pt["validation"])

        trainer.train()

        # save model
        model_pegasus.save_pretrained(os.path.join(self.config.root_dir, "pegasus-samsum-model"))

        # save tokenizer
        tokenizer.save_pretrained(os.path.join(self.config.root_dir, "tokenizer"))


from sphfile import SPHFile
import glob
import os
""""Convert SPH file to wav"""

dialects_path = "/home/marcin/Pobrane/TIMIT"
root_dir = os.path.join(dialects_path, '**/*.WAV')
wav_files = glob.glob(root_dir, recursive=True)

for wav_file in wav_files:
    sph = SPHFile(wav_file)
    txt_file = ""
    txt_file = wav_file[:-3] + "TXT"

    f = open(txt_file,'r')
    for line in f:
        words = line.split(" ")
        start_time = (int(words[0])/16000)
        end_time = (int(words[1])/16000)
    print("writing file ", wav_file)
    sph.write_wav(wav_file.replace(".WAV",".wav"),start_time,end_time)


# 题目
# 求1+2+3+...+n，要求不能使用乘除法、for、while、if、else、switch、case等关键字及条件判断语句（A?B:C）。

# 思路
# 利用递归当作计算结果

# -*- coding:utf-8 -*-
class Solution:
    def Sum_Solution(self, n):
        # write code here
        if n==0:
            return 0
        return self.Sum_Solution(n-1)+n

if __name__=='__main__':
    n=6
    solution=Solution()
    ans=solution.Sum_Solution(n)
    print(ans)


#!/usr/bin/env python
import os
activate_this = os.path.expanduser("~/PycharmProjects/DefensicsAutomation/venv/bin/activate_this.py")
execfile(activate_this, dict(__file__=activate_this))

from datetime import datetime
import CheckAndSendResults
import GlobalVariables
import subprocess
import os

def send_daily_results(path, mail_title):
    """
    This function will send Daily results (if have)

    :param path: path to results directory
    :param mail_title: title of the mail to send
    """
    CheckAndSendResults.send_results(path, mail_title)

def check_run_currently_running(running_lines):
    i = 0
    detect_running = "ps -aux | grep java | grep testplan | grep {}.testplan | tr -s ' ' | cut -d ' ' -f 2"

    for line in running_lines:
        my_list = line.replace('\n', '').split(',')
        if i == 0 or len(my_list) < 2:
            i = 1
            continue
        name = my_list[1]

        proc = subprocess.Popen(detect_running.format(name), stdout=subprocess.PIPE, shell=True)
        output = proc.stdout.read().replace('\n', ',').split(',')

        if len(output) > 2:
            return name

    return None

def get_running_dir(list_dir):
    path_to_all_results = GlobalVariables.part_of_path_to_results
    list_results = []

    for item in list_dir:
        location_running = path_to_all_results + item + '/ListOfRunningRunsDetails.txt'

        if os.path.exists(location_running):
            list_results.append(item)

    # sort list of results
    list_results.sort()
    name_last_run = list_results[len(list_results) - 1]
    running_lines = open(path_to_all_results + name_last_run + '/ListOfRunningRunsDetails.txt', 'r').readlines()
    waiting_lines = open(path_to_all_results + name_last_run + '/ListOfWaitingRunsDetails.txt', 'r').readlines()

    if len(running_lines) > 1 or len(waiting_lines) > 0:
        found_running = check_run_currently_running(running_lines)
        if found_running is not None:
            print("Note: There are runs that run in background")
            return name_last_run

    return ''

def get_path_last_run():
    """
    This function will return the last results directory that was created today.
    """
    path_to_all_results = GlobalVariables.part_of_path_to_results
    list_today_results = []

    date = str(datetime.now())
    date = date[:date.find(' ')]

    for item in os.listdir(path_to_all_results):
        if date in item:
            list_today_results.append(item)

    if len(list_today_results) == 0:
        # if no run started today, check if run of other day still running
        path_to_last_result_today = get_running_dir(os.listdir(path_to_all_results))
        if path_to_last_result_today == '':
            print("\n=====No Automated Run ran today, and no one is running=====\n")
            exit()
    else:
        # sort list of results
        list_today_results.sort()

        path_to_last_result_today = list_today_results[len(list_today_results) - 1]

    print('\nLast result today-> ' + path_to_last_result_today)
    return path_to_all_results + '/' + path_to_last_result_today

if __name__ == '__main__':
    additional_txt_tittle = "Daily Results"

    path_to_current_run = get_path_last_run()
    send_daily_results(path_to_current_run, additional_txt_tittle)