#!/usr/bin/env python
# coding: utf-8

# # Writing Reusable Code Using Functions in Python
#
# ### Part 4 of "A Gentle Introduction to Programming with Python"
#
# This tutorial is the fourth in a series on introduction to programming using the Python language. These tutorials take a practical coding-based approach, and the best way to learn the material is to execute the code and experiment with the examples. Check out the full series here:
#
# 1. [First Steps with Python and Jupyter](https://jovian.ml/aakashns/first-steps-with-python)
# 2. [A Quick Tour of Variables and Data Types](https://jovian.ml/aakashns/python-variables-and-data-types)
# 3. [Branching using Conditional Statements and Loops](https://jovian.ml/aakashns/python-branching-and-loops)
# 4. [Writing Reusable Code Using Functions](https://jovian.ml/aakashns/python-functions-and-scope)
# 5. [Reading from and Writing to Files](https://jovian.ml/aakashns/python-os-and-filesystem)
# 6. [Object Oriented Programming with Classes](https://jovian.ml/aakashns/python-object-oriented-programming)
#
#

# ## How to run the code
#
# This tutorial hosted on [Jovian.ml](https://www.jovian.ml), a platform for sharing data science projects online. You can "run" this tutorial and experiment with the code examples in a couple of ways: *using free online resources* (recommended) or *on your own computer*.
#
# >  This tutorial is a [Jupyter notebook](https://jupyter.org) - a document made of "cells", which can contain explanations in text or code written in Python. Code cells can be executed and their outputs e.g. numbers, messages, graphs, tables, files etc. can be viewed within the notebook, which makes it a really powerful platform for experimentation and analysis. Don't afraid to experiment with the code & break things - you'll learn a lot by encoutering and fixing errors. You can use the "Kernel > Restart & Clear Output" menu option to clear all outputs and start again from the top of the notebook.
#
# ### Option 1: Running using free online resources (1-click, recommended)
#
# The easiest way to start executing this notebook is to click the "Run" button at the top of this page, and select "Run on Binder". This will run the notebook on [mybinder.org](https://mybinder.org), a free online service for running Jupyter notebooks. You can also select "Run on Colab" or "Run on Kaggle", but you'll need to create an account on [Google Colab](https://colab.research.google.com) or [Kaggle](https://kaggle.com) to use these platforms.
#
#
# ### Option 2: Running on your computer locally
#
# You'll need to install Python and download this notebook on your computer to run in locally. We recommend using the [Conda](https://docs.conda.io/en/latest/) distribution of Python. Here's what you need to do to get started:
#
# 1. Install Conda by [following these instructions](https://conda.io/projects/conda/en/latest/user-guide/install/index.html). Make sure to add Conda binaries to your system `PATH` to be able to run the `conda` command line tool from your Mac/Linux terminal or Windows command prompt.
#
#
# 2. Create and activate a [Conda virtual environment](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) called `intro-to-python` which you can use for this tutorial series:
# ```
# conda create -n intro-to-python -y python=3.8
# conda activate intro-to-python
# ```
# You'll need to create the environment only once, but you'll have to activate it every time want to run the notebook. When the environment is activated, you should be able to see a prefix `(intro-to-python)` within your terminal or command prompt.
#
#
# 3. Install the required Python libraries within the environmebt by the running the following command on your  terminal or command prompt:
# ```
# pip install jovian jupyter numpy pandas matplotlib seaborn --upgrade
# ```
#
# 4. Download the notebook for this tutorial using the `jovian clone` command:
# ```
# jovian clone aakashns/python-functions-and-scope
# ```
# The notebook is downloaded to the directory `python-functions-and-scope`.
#
#
# 5. Enter the project directory and start the Jupyter notebook:
# ```
# cd python-functions-and-scope
# jupyter notebook
# ```
#
# 6. You can now access Jupyter's web interface by clicking the link that shows up on the terminal or by visiting http://localhost:8888 on your browser. Click on the notebook `python-functions-and-scope.ipynb` to open it and run the code. If you want to type out the code yourself, you can also create a new notebook using the "New" button.
#
#

# ## Creating and using functions
#
# A function is a reusable set of instructions. A function takes one or more inputs, performs certain operations, and often returns an output. Python provides many in-built functions like `print`, and also allows you to define your own functions.

# In[1]:

today = "Saturday"
print("Today is", today)

# You can define a new function using the `def` keyword.

# In[2]:

def say_hello():
    print('Hello there!')
    print('How are you?')

# Note the round brackets or parantheses `()` and colon `:` after the function's name. Both are essential parts of the syntax for defining a function. The *body* of the function can contain one or more statements which are to be executed when the function is called. Simlar to conditional statements and loops, the statements must be indented by 4 spaces.
#
# The statements inside a function's body are not executed when a function is defined. To execute the statements, we need to *call* or *invoke* the function.

# In[3]:

say_hello()

# ### Function arguments
#
# Functions can also accept one or more values as *inputs* (also knows as *arguments* or *parameters*). Arguments help us write flexible functions which can perform the same operation on different values. Further, functions can also return a value as a result using the `return` keyword, which can be stored in a variable or used in other expressions.
#
# Here's a function that filters out the even numbers from a list.

# In[4]:

def filter_even(number_list):
    result_list = []
    for number in number_list:
        if number % 2 == 0:
            result_list.append(number)
    return result_list

# Can you understand what the function does by looking at the code? If not, try to execute each line of the function within a code cell with a list of numbers as `number_list`.

# In[5]:

even_list = filter_even([1, 2, 3, 4, 5, 6, 7])

# In[6]:

even_list

# ## Writing great functions in Python
#
# As a programmer, you will spend most of your time writing and using functions, and Python offers many features to make your functions powerful and flexible. Let's explore some of these by solving a problem:
#
# > Radha is planning to buy a house that costs `$1,260,000`. She considering two options to finance her purchase:
# >
# > * Option 1: Make an immediate down payment of `$300,000`, and take loan 8-year loan with an interest rate of 10% (compounded monthly) for the remaining amount.
# > * Option 2: Take a 10-year loan with an interest rate of 8% (compounded monthly) for the entire amount.
# >
# > Both these loans have to paid back in equal monthly installments (EMIs). Which loan has a lower EMI among the two?
#
#
# Since we need to compare the EMIs for two loan options, it might be helpful to define a function to calculate the EMI for a loan, given inputs like the cost of the house, the down payment, duration of the loan, rate of interest etc. We'll build this function step by step.
#
# To begin, let's write a simple function that calculates the EMI on the entire cost of the house, assuming that the loan has to be paid back in one year, and there is no interest or down payment.

# In[7]:

def loan_emi(amount):
    emi = amount / 12
    print('The EMI is ${}'.format(emi))

# In[8]:

loan_emi(1260000)

# ### Local variables and scope
#
# Let's add a second argument to account for the duration of the loan, in months.

# In[9]:

def loan_emi(amount, duration):
    emi = amount / duration
    print('The EMI is ${}'.format(emi))

# Note that the variable `emi` defined inside the function is not accessible outside the function. The same is true for the parameters `amount` and `duration`. These are all *local variables* that lie within the *scope* of the function.
#
# > **Scope**: Scope refers to the region within the code where a certain variable is visible. Every function (or class definition) defines a scope within Python. Variables defined in this scope are called *local variables*. Variables that are available everywhere are called *global variables*. Scope rules allow you to use the same variable names in different functions without sharing values from one to the other.

# In[10]:

emi

# In[11]:

amount

# In[12]:

duration

# We can now compare a 6-year loan vs. a 10-year loan (assuming no down payment or interest).

# In[13]:

loan_emi(1260000, 8*12)

# In[14]:

loan_emi(1260000, 10*12)

# ### Return values
#
# As you might expect, the EMI for the 6-year loan is higher compared to the 10-year loan. Right now we're printing out the result, but it would be better to return it and store the results in variables for easier comparison. We can do this using the `return` statement

# In[15]:

def loan_emi(amount, duration):
    emi = amount / duration
    return emi

# In[16]:

emi1 = loan_emi(1260000, 8*12)

# In[17]:

emi2 = loan_emi(1260000, 10*12)

# In[18]:

emi1

# In[19]:

emi2

# ### Optional arguments
#
# Let's now add another argument to account for the immediate down payment. We'll make this an *optional argument*, with a default value of 0.

# In[20]:

def loan_emi(amount, duration, down_payment=0):
    loan_amount = amount - down_payment
    emi = loan_amount / duration
    return emi

# In[21]:

emi1 = loan_emi(1260000, 8*12, 3e5)

# In[22]:

emi1

# In[23]:

emi2 = loan_emi(1260000, 10*12)

# In[24]:

emi2

# Next, let's add the interest calculation into the function. Here's the formula used to calculate the EMI for a loan:
#
# <img src="https://i.imgur.com/iKujHGK.png" style="width:240px">
#
# where:
#
# * `P` is the loan amount (principal)
# * `n` is the no. of months
# * `r` is the rate of interest per month
#
# The derivation of this forumula is beyond the scope of this tutorial. See this video for an explanation: https://youtu.be/Coxza9ugW4E

# In[25]:

def loan_emi(amount, duration, rate, down_payment=0):
    loan_amount = amount - down_payment
    emi = loan_amount * rate * ((1+rate)**duration) / (((1+rate)**duration)-1)
    return emi

# Note that while defining the function, required arguments like `cost`, `duration` and `rate` must appear before optional arguments like `down_payment`.
#
# Let's calculate the EMI for Option 1

# In[26]:

loan_emi(1260000, 8*12, 0.1/12, 3e5)

# While calculating the EMI for Option 2, we need not include the `down_payment` argument.

# In[27]:

loan_emi(1260000, 10*12, 0.08/12)

# ### Named arguments
#
# Invoking a function with many arguments can often get confusing, and is prone to human errors. Python provides the option of invoking functions with *named* arguments, for better clarity. Function invocation can also be split into multiple lines.

# In[28]:

emi1 = loan_emi(
    amount=1260000,
    duration=8*12,
    rate=0.1/12,
    down_payment=3e5
)

# In[29]:

emi1

# In[30]:

emi2 = loan_emi(amount=1260000, duration=10*12, rate=0.08/12)

# In[31]:

emi2

# ### Modules and library functions
#
# We can already see that the EMI for Option 1 seems to be lower than the EMI for Option 2. However, it would be nice to round up the amount to full dollars, rather than including digits afer the decimal. To achieve this, we might want to write a function which can take a number and round it up to the next integer (e.g. 1.2 is rounded up to 2). That would be a good exercise to try out!
#
# However, since rounding numbers is a fairly common operation, Python provides a function for it (along with thousands of other functions) as part of the [Python Standard Library](https://docs.python.org/3/library/). Functions are organized into *modules*, which need to imported in order to use the functions they contain.
#
# > **Modules**: Modules are files containing Python code (variables, functions, classes etc.). They provide a way of organizing the code for large Python projects into files and folders. The key benefit offered by modules is *namespaces* - a module or a specific function/class/variable from a module has to imported before it can be used within a Python script or notebook. This provides *encapsulation* and avoid naming conflicts between your code vs. a module, or across modules.
#
# For rounding up our EMI amounts, we can use the `ceil` function (short for *ceiling*) from the `math` module. Let's import the module and use it to round up the number 1.2 .

# In[32]:

import math

# In[33]:

help(math.ceil)

# In[34]:

math.ceil(1.2)

# Let's now use the `math.ceil` function within the `home_loan_emi` function to round up the EMI amount.
#
# > Using function to build other functions is a great way to reuse code and implement complex business logic while still keeping the code small, understandable and manageable. Ideally, one a fuction should do one thing, and one thing only. If you find yourself doing too many things within a single function, you should consider splitting it into 2 or more smaller, independent functions. As a rule of thumb, try to limit your functions to 10 lines of code or less. Good programmers always write small, simple and readable functions.
#
#

# In[35]:

def loan_emi(amount, duration, rate, down_payment=0):
    loan_amount = amount - down_payment
    emi = loan_amount * rate * ((1+rate)**duration) / (((1+rate)**duration)-1)
    emi = math.ceil(emi)
    return emi

# In[36]:

emi1 = loan_emi(
    amount=1260000,
    duration=8*12,
    rate=0.1/12,
    down_payment=3e5
)

# In[37]:

emi1

# In[38]:

emi2 = loan_emi(amount=1260000, duration=10*12, rate=0.08/12)

# In[39]:

emi2

# Let's compare the EMIs and display a message for the option with the lower EMI.

# In[40]:

if emi1 < emi2:
    print("Option 1 has the lower EMI: ${}".format(emi1))
else:
    print("Option 2 has the lower EMI: ${}".format(emi2))

# ### Reusing and improving functions
#
# Now we know for cetain that "Option 1" has the lower EMI among the two options. But what's even better is that we now have a handy function `loan_emi` that can be used to solve many other similar problems with just a few lines of code. Let's try it with a couple more problems.

# > Q: Shaun is currenly paying back a home loan for a house a few years go. The cost of the house was `$800,000`. Shaun made a down payment of `25%` of the cost, and financed the remaining amount using a 6-year loan with an interest rate of `7%` per annum (compounded monthly). Shaun is now buying a car worth `$60,000`, which he is planning to finance using a 1-year loan with an interest rate of `12%` per annum. Both loans are paid back in EMIs. What is the total monthly payment Shaun makes towards loan repayment?
#
# This question is now straightforward to solve, using the `loan_emi` function we've already defined.

# In[41]:

cost_of_house = 800000
home_loan_duration = 6*12 # months
home_loan_rate = 0.07/12 # monthly
home_down_payment = .25 * 800000

emi_house = loan_emi(amount=cost_of_house,
                     duration=home_loan_duration,
                     rate=home_loan_rate,
                     down_payment=home_down_payment)

emi_house

# In[42]:

cost_of_car = 60000
car_loan_duration = 1*12 # months
car_loan_rate = .12/12 # monthly

emi_car = loan_emi(amount=cost_of_car,
                   duration=car_loan_duration,
                   rate=car_loan_rate)

emi_car

# In[43]:

print("Shaun makes a total monthly payment of ${} towards loan repayments.".format(emi_house+emi_car))

# ### Exceptions and `try`-`except`
#
# > Q: If you borrow `$100,000` using a 10-year loan with an interest rate of 9% per annum, what is the total amount you end up paying as interest?
#
# One way to solve this problem is to compare the EMIs for two loans: one with the given rate of interest, and another with a 0% rate of interest. The total interest paid is then simply the sum of monthly differences over the duration of the loan.

# In[44]:

emi_with_interest = loan_emi(amount=100000, duration=10*12, rate=0.09/12)
emi_with_interest

# In[45]:

emi_without_interest = loan_emi(amount=100000, duration=10*12, rate=0./12)
emi_without_interest

# Something seems to have gone wrong! If you look at the error message above carefully, Python tells us exactly what is gone wrong. Python *throws* a `ZeroDivisionError` with a message indicating that we're trying to divide a number by zero. This is an *exception* that stops further execution of the program.
#
# > **Exception**: Even if a statement or expression is syntactically correct, it may cause an error when an attempt is made to execute it. Errors detected during execution are called exceptions. We refer to exceptions as being typically stop further execution of the program, unless they are handled within the program using `try`-`except` statements.
#
# Python provides many built-in exceptions that are *thrown* when built-in operators, functions or methods are used in an incorrect manner: https://docs.python.org/3/library/exceptions.html#built-in-exceptions . You can also define your own custom exception by extending `Exception` class (more on that later).
#
# You can use the `try` and `except` statements to *handle* and exception. Here's an example:

# In[46]:

try:
    print("Now computing the result..")
    result = 5 / 0
    print("Computation was completed successfully")
except ZeroDivisionError:
    print("Failed to compute result because you were trying to divide by zero")
    result = None

print(result)

# When an exception occurs in the code inside a `try` block, the rest of the statements in the block are skipped, and `except` statement is executed. If the type of exception throw matches the type of exception being handled by the `except` statement, then the code inside the `except` block is executed and the program execution then returns to the normal flow.
#
# You can also handle more than one type of exception using multiple `except` statements. Learn more about exceptions here: https://www.w3schools.com/python/python_try_except.asp .
#
# Let's enhance the `loan_emi` function to use `try`-`execpt` to handle the scenario where the rate of intersest is 0%. It's common practice to make changes/enhancements to functions over time, as new scenarios and use cases come up. It makes functions more flexible & powerful.

# In[47]:

def loan_emi(amount, duration, rate, down_payment=0):
    loan_amount = amount - down_payment
    try:
        emi = loan_amount * rate * ((1+rate)**duration) / (((1+rate)**duration)-1)
    except ZeroDivisionError:
        emi = loan_amount / duration
    emi = math.ceil(emi)
    return emi

# We can use the updated `loan_emi` function to solve our problem.
#
# > Q: If you borrow `$100,000` using a 10-year loan with an interest rate of 9% per annum, what is the total amount you end up paying as interest?
#
#

# In[48]:

emi_with_interest = loan_emi(amount=100000, duration=10*12, rate=0.09/12)
emi_with_interest

# In[49]:

emi_without_interest = loan_emi(amount=100000, duration=10*12, rate=0)
emi_without_interest

# In[50]:

total_interest = (emi_with_interest - emi_without_interest) * 10*12

# In[51]:

print("The total interest paid is ${}.".format(total_interest))

# ### Documenting functions using Docstrings
#
# We can add some documentation within our function using a *docstring*. A docstring is simply a string that appears as the first statement within the function body, and is used by the `help` function. A good docstring describes what the function does, and provides some explanation about the arguments.

# In[52]:

def loan_emi(amount, duration, rate, down_payment=0):
    """Calculates the equal montly installment (EMI) for a loan.

    Arguments:
        amount - Total amount to be spent (loan + down payment)
        duration - Duration of the loan (in months)
        rate - Rate of interest (monthly)
        down_payment (optional) - Optional intial payment (deducted from amount)
    """
    loan_amount = amount - down_payment
    try:
        emi = loan_amount * rate * ((1+rate)**duration) / (((1+rate)**duration)-1)
    except ZeroDivisionError:
        emi = loan_amount / duration
    emi = math.ceil(emi)
    return emi

# In the docstring above, we've provided some additional information that the `duration` and `rate` are both measured in months. You might even consider naming the arguments `duration_months` and `rate_monthly`, to avoid any confusion whatsoever. Can you think of some other ways in which the function can be improved?

# In[53]:

help(loan_emi)

# ## Save and upload your notebook

# Whether you're running this Jupyter notebook on an online service like Binder or on your local machine, it's important to save your work from time, so that you can access it later, or share it online. You can upload this notebook to your [Jovian.ml](https://jovian.ml) account using the `jovian` Python library.

# In[54]:

# Instal the library
get_ipython().system('pip install jovian --upgrade --quiet')

# In[55]:

# Import the jovian module
import jovian

# In[ ]:

jovian.commit(project='python-functions-and-scope', environment=None)

# ## Exercise - Data Analysis for Vacation Planning
#
# You're planning a leisure trip (vacation) and you need to decide which city you want to visit. You have shortlisted 4 cities, and identified the cost of the return flight, daily hotel cost and a weekly car rental cost (a car has to be rented for full weeks, even if you return the car before a week ends).
#
#
# | City | Return Flight (`$`) | Hotel per day (`$`) | Weekly Car Rental  (`$`) |
# |------|--------------------------|------------------|------------------------|
# | Paris|       200                |       20         |          200           |
# | London|      250                |       30         |          120           |
# | Dubai|       370                |       15         |          80           |
# | Mumbai|      450                |       10         |          70           |
#
#
# Answer the following questions using the data above:
#
# 1. If you're planning a 1-week long trip, which city should you visit to spend the least amount of money?
# 2. How does the answer to the previous question change if you change the duration of the trip to 4 days, 10 days or 2 weeks?
# 3. If your total budget for the trip is `$1000`, which city should you visit to maximize the duration of your trip? Which city should you visit if you want to minimize the duration?
# 4. How does the answer to the previous question change if your budget is `$600`, `$2000` or `$1500`?
#
# *Hint: To answer these questions, it will help to define a function `cost_of_trip` with relevant inputs like flight cost, hotel rate, car rental rate and duration of the trip. You may find the `math.ceil` function useful for calculating the total cost of car rental.*

# In[ ]:

# Use these cells to answer the question - build the function step-by-step

# In[ ]:

# In[ ]:

# In[ ]:

# ## Summary and Further Reading
#
# With this we complete our discussion of functions in Python. We've covered the following topics in this tutorial:
#
# * Creating and using functions
# * Functions with one or more arguments
# * Local variables and scope
# * Returning values using `return`
# * Using default arguments to make a function flexible
# * Using named arguments while invoking a function
# * Importing modules and using library functions
# * Reusing and improving functions to handle new use cases
# * Handling exceptions with `try`-`except`
# * Documenting functions using docstrings
#
# This is by no means an exhaustive or comprehensive tutorial on functions in Python. Here are few more topics to learn about:
#
# * Functions with an arbitrary number of arguments using (`*args` and `**kwargs`)
# * Defining functions inside functions (and closures)
# * A function that invokes itself (recursion)
# * Functions that accept other functions as arguments or return other functions
# * Functions that enhance other functions (decorators)
#
# Following are some resources to learn about more functions in Python:
#
# * Python Tutorial at W3Schools: https://www.w3schools.com/python/
# * Practical Python Programming: https://dabeaz-course.github.io/practical-python/Notes/Contents.html
# * Python official documentation: https://docs.python.org/3/tutorial/index.html
#
# You are ready to move on to the next tutorial: ["Reading from and writing to files using Python"](https://jovian.ml/aakashns/python-os-and-filesystem).

# In[ ]:


import inspect

from sybil_engine.contract.erc20contract import Erc20Contract
from sybil_engine.contract.weth import WETH
from sybil_engine.data.tokens import get_tokens_for_chain
from sybil_engine.domain.balance.balance import Erc20Balance, WETHBalance

class Erc20Token:
    def __init__(self, chain, token, web3):
        self.chain = chain
        self.token = token
        self.web3 = web3
        self.erc20_contract = Erc20Contract(get_tokens_for_chain(self.chain)[self.token], self.web3)

    def balance(self, account):
        return Erc20Balance(
            self.erc20_contract.balance_of(account),
            self.chain,
            self.token,
            decimal=self.erc20_contract.decimals()
        )

    def approve(self, account, contract_on_approve):
        caller_frame = inspect.currentframe().f_back
        caller_function = caller_frame.f_code.co_name
        if caller_function == 'execute_transaction_internal':
            original_function = self.erc20_contract.approve.__wrapped__ if hasattr(self.erc20_contract.approve,
                                                                                "__wrapped__") else self.erc20_contract.approve
            return original_function(self.erc20_contract, account, contract_on_approve)
        else:
            return self.erc20_contract.approve(account, contract_on_approve)

    def allowance(self, account, allowance_contract):
        return self.erc20_contract.allowance(account, allowance_contract)

class WETHToken:
    def __init__(self, chain, web3):
        self.chain = chain
        self.web3 = web3
        self.weth_contract = WETH(get_tokens_for_chain(self.chain)['WETH'], self.web3)

    def balance(self, account):
        return WETHBalance(self.weth_contract.balance_of(account), self.chain)


import tensorflow as tf
from tensorflow import keras
import attention
class feed_forward(keras.Model):
	def __init__(self,max_seq_len,d_model):
		super(feed_forward,self).__init__()
		self.d_model=d_model
		self.max_seq_len=max_seq_len
		self.weights_encoder=self.add_weight(
			name='final_weights',
			shape=(self.d_model,self.d_model),
			initializer='random_normal',
			trainable=True
			)
		self.bias=self.add_weight(
			name='bias1',
			shape=(self.max_seq_len,self.d_model),
			initializer='random_normal',
			trainable=True
			)
	def call(self,data):
		return tf.matmul(data,self.weights_encoder)+self.bias

class encoder(keras.Model):
	def __init__(self,number_heads,max_seq_len,d_model,expected_len):
		super(encoder,self).__init__()
		self.number_heads=number_heads
		self.max_seq_len=max_seq_len
		self.d_model=d_model
		self.expected_len=expected_len
		self.mha=attention.MultiHeadAttention(number_heads=self.number_heads,max_seq_len=self.max_seq_len,d_model=self.d_model,expected_len=self.expected_len)
		self.normalise1=tf.keras.layers.LayerNormalization(epsilon=1e-6)
		self.normalise2=tf.keras.layers.LayerNormalization(epsilon=1e-6)
		self.drop1=tf.keras.layers.Dropout(0.1)
		self.forward=feed_forward(max_seq_len,d_model)

	def call(self,data):
		attention_output=self.mha(data)
		attention_output=self.drop1(attention_output)
		first_out=self.normalise1(attention_output+data)
		first_out1=self.forward(first_out)
		out=self.normalise2(first_out1+first_out)
		return out


from helpers.func import (
    resJson,
    resType,
    exceptionRaise,
    validate_list
)
from helpers.validator import paginateValidator, postProdAttributeValueValidator, updateProdAttributeValueValidator
from helpers.query import (
    paginatedQuery,
    insertQuery,
    softDelbyId,
    findRecordById,
    findRecordByColumn,
    findRecordByColumnCron,
    updateById
)
from sanic.log import logger
from sanic import Blueprint
from models.product_attribute_value import ProductAttributeValue
from utils.auth import protected
from sqlalchemy import update
from odoo.main import get_prod_attr_value

import arrow

# -----------------
# API Class Section
# -----------------

class ProductAttributeValueController():
    p = Blueprint('prod_attribute_value', url_prefix='/')

    @p.get("/prod-attribute-values")
    @protected
    async def getProdAttributeValues(request):
        try:
            session = request.ctx.session
            params = request.args
            # Input validation
            [valid, error] = paginateValidator(params)
            if not valid:
                return resJson(resType.INVALID_PARAMS, error, len(error))

            async with session.begin():
                size = int(params.get('pageSize', 10))
                page = int(params.get('page', 1))
                sort = str(params.get('sortParam', 'created_at'))
                order = str(params.get('sortBy', 'DESC'))
                select_items: any = [
                    ProductAttributeValue.id,
                    ProductAttributeValue.odoo_id,
                    ProductAttributeValue.name,
                    ProductAttributeValue.display_name,
                    ProductAttributeValue.ptav_active,
                    ProductAttributeValue.product_attribute_value_id,
                    ProductAttributeValue.attribute_line_id,
                    ProductAttributeValue.price_extra,
                    ProductAttributeValue.exclude_for,
                    ProductAttributeValue.product_tmpl_id,
                    ProductAttributeValue.attribute_id,
                    ProductAttributeValue.ptav_product_variant_ids,
                    ProductAttributeValue.is_custom,
                    ProductAttributeValue.display_type,
                    ProductAttributeValue.created_at
                ]

                [stmt, count_] = paginatedQuery(
                    ProductAttributeValue, sort, order, select_items, size, page)
                result = await session.execute(stmt)
                attribute_values = result.all()

                result_dict = [dict(attribute_value)
                               for attribute_value in attribute_values]
                total_ = await session.execute(count_)
                total = total_.scalar()

            return resJson(resType.OK, result_dict, total)
        except:
            exceptionRaise('getProdAttributeValues')

    @p.get("/prod-attribute-value/<pk_:uuid>")
    @protected
    async def getProdAttributeValue(request, pk_):
        try:
            session = request.ctx.session
            async with session.begin():
                attribute_value = await findRecordById(session, ProductAttributeValue, pk_)

            if not attribute_value:
                return resJson(resType.NO_RECORD)

            return resJson(resType.OK, attribute_value.to_dict())
        except:
            exceptionRaise('getProdAttributeValue')

    # MARK: Support scalar and multi input
    @p.post("/prod-attribute-value")
    @protected
    async def createProdAttributeValue(request):
        try:
            session = request.ctx.session
            body = request.json

            async with session.begin():
                if validate_list(body):
                    body_records = body
                else:
                    body_records = [body]

                # Input validation
                [valid, error] = postProdAttributeValueValidator(body_records)
                if not valid:
                    return resJson(resType.INVALID_PARAMS, error, len(error))

                output_list = []

                [new_list, update_list, redundant_ids] = await insertOrUpdate(session, body_records)
                if (not update_list) and (not redundant_ids):
                    # Post new record
                    result = await insertQuery(session, ProductAttributeValue, new_list)
                else:
                    return resJson(resType.DUPLICATE, redundant_ids, len(redundant_ids))

                for u in result:
                    output_list.append(dict(u))

            return resJson(resType.OK, output_list, len(output_list))
        except:
            exceptionRaise('createProdAttributeValue')

    @p.delete("/prod-attribute-value/<pk_:uuid>")
    @protected
    async def destroyProdAttributeValue(request, pk_):
        try:
            session = request.ctx.session
            resMsg = resType.SUCCESS_DEL
            async with session.begin():
                attribute_value = await findRecordById(session, ProductAttributeValue, pk_)
                if not attribute_value:
                    return resJson(resType.NO_RECORD)

                destroy = await softDelbyId(session, ProductAttributeValue, pk_)
                if not destroy:
                    resMsg = resType.FAIL_DELETE

            return resJson(resMsg, destroy)
        except:
            exceptionRaise('destroyProdAttributeValue')

    @p.put("/prod-attribute-value/<pk_:uuid>")
    @protected
    async def updateProdAttributeValue(request, pk_):
        try:
            session = request.ctx.session
            b = request.json
            # Input validation
            [valid, error] = updateProdAttributeValueValidator(b)
            if not valid:
                return resJson(resType.INVALID_PARAMS, error, len(error))

            resMsg = resType.SUCCESS_UPD
            async with session.begin():
                attribute_value = await findRecordById(session, ProductAttributeValue, pk_)
                if not attribute_value:
                    return resJson(resType.NO_RECORD)

                id = b.get('id', None)
                name = b.get('name', None)
                display_name = b.get('display_name', None)
                ptav_active = b.get('ptav_active', None)
                product_attribute_value_id = b.get('product_attribute_value_id', None)
                attribute_line_id = b.get('attribute_line_id', None)
                price_extra = b.get('price_extra', None)
                exclude_for = b.get('exclude_for', None)
                product_tmpl_id = b.get('product_tmpl_id', None)
                attribute_id = b.get('attribute_id', None)
                ptav_product_variant_ids = b.get('ptav_product_variant_ids', None)
                is_custom = b.get('is_custom', None)
                display_type = b.get('display_type', None)
                create_date = b.get('create_date', None)

                w_record = {}
                date_ = arrow.get(str(create_date)).datetime

                if id != attribute_value.odoo_id:
                    w_record['odoo_id'] = int(id)
                if name != attribute_value.name:
                    w_record['name'] = name
                if display_name != attribute_value.display_name:
                    w_record['display_name'] = display_name
                if bool(ptav_active) != attribute_value.ptav_active:
                    w_record['ptav_active'] = bool(ptav_active)
                if int(product_attribute_value_id[0]) != attribute_value.product_attribute_value_id:
                    w_record['product_attribute_value_id'] = int(
                        product_attribute_value_id[0])
                if int(attribute_line_id[0]) != attribute_value.attribute_line_id:
                    w_record['attribute_line_id'] = int(attribute_line_id[0])
                if float(price_extra) != attribute_value.price_extra:
                    w_record['price_extra'] = float(price_extra)
                if list(exclude_for) != attribute_value.exclude_for:
                    w_record['exclude_for'] = list(
                        exclude_for)
                if int(product_tmpl_id[0]) != attribute_value.product_tmpl_id:
                    w_record['product_tmpl_id'] = int(product_tmpl_id[0])
                if int(attribute_id[0]) != attribute_value.attribute_id:
                    w_record['attribute_id'] = int(attribute_id[0])
                if list(ptav_product_variant_ids) != attribute_value.ptav_product_variant_ids:
                    w_record['ptav_product_variant_ids'] = list(
                        ptav_product_variant_ids)
                if bool(is_custom) != attribute_value.is_custom:
                    w_record['is_custom'] = bool(is_custom)
                if display_type != attribute_value.display_type:
                    w_record['display_type'] = display_type
                if date_.replace(tzinfo=None) != attribute_value.created_at:
                    w_record['created_at'] = date_.replace(tzinfo=None)

                if len(w_record) < 1:
                    return resJson(resType.NO_UPD, {})

                setAttr = await updateById(session, ProductAttributeValue, pk_, w_record)
                if not setAttr:
                    resMsg = resType.FAIL_UPD

            return resJson(resMsg, setAttr)
        except:
            exceptionRaise('updateProdAttributeValue')

    @p.patch("/prod-attribute-value")
    @protected
    async def addOrUpdateProdAttributeValue(request):
        try:
            session = request.ctx.session
            body = request.json

            async with session.begin():
                if validate_list(body):
                    body_records = body
                else:
                    body_records = [body]

                # Input validation
                [valid, error] = postProdAttributeValueValidator(body_records)
                if not valid:
                    return resJson(resType.INVALID_PARAMS, error, len(error))

                output_list = []

                [new_list, update_list, redundant_ids] = await insertOrUpdate(session, body_records)
                if new_list:
                    # Post new record
                    result = await insertQuery(session, ProductAttributeValue, new_list)

                    for u in result:
                        output_list.append(u.id)

                if update_list:
                    result = await bulkUpdateQuery(session, ProductAttributeValue, redundant_ids, update_list)

                    for u in result:
                        output_list.append(u)

                if not output_list:
                    return resJson(resType.NO_UPD, {})

            return resJson(resType.OK, output_list, len(output_list))
        except:
            exceptionRaise('addOrUpdateProdAttributeValue')

# -----------------
# functions section
# -----------------
async def insertOrUpdate(session, body, bg=False):
    try:
        new_list, update_list, redundant_ids = [], [], []
        # Sort body data by id
        body.sort(reverse=False, key=lambda e: e['id'])

        for b in body:
            id = b.get('id', None)
            name = b.get('name', None)
            display_name = b.get('display_name', None)
            ptav_active = b.get('ptav_active', None)
            product_attribute_value_id = b.get('product_attribute_value_id', None)
            attribute_line_id = b.get('attribute_line_id', None)
            price_extra = b.get('price_extra', None)
            exclude_for = b.get('exclude_for', None)
            product_tmpl_id = b.get('product_tmpl_id', None)
            attribute_id = b.get('attribute_id', None)
            ptav_product_variant_ids = b.get('ptav_product_variant_ids', None)
            is_custom = b.get('is_custom', None)
            display_type = b.get('display_type', None)
            create_date = b.get('create_date', None)
            w_record = {}

            if not bg:
                attribute_value = await findRecordByColumn(session, ProductAttributeValue, ProductAttributeValue.odoo_id, int(id), False)
            else:
                attribute_value = await findRecordByColumnCron(session, ProductAttributeValue, ProductAttributeValue.odoo_id, int(id), False)
            if not attribute_value:
                # register record
                date_ = arrow.get(str(create_date)).datetime
                w_record = {
                    "odoo_id": int(id),
                    "name": str(name),
                    "display_name": str(display_name),
                    "ptav_active": bool(ptav_active),
                    "product_attribute_value_id": int(product_attribute_value_id[0]),
                    "attribute_line_id": int(attribute_line_id[0]),
                    "price_extra": float(price_extra),
                    "exclude_for": list(exclude_for),
                    "product_tmpl_id": int(product_tmpl_id[0]),
                    "attribute_id": int(attribute_id[0]),
                    "ptav_product_variant_ids": list(ptav_product_variant_ids),
                    "is_custom": bool(is_custom),
                    "display_type": str(display_type),
                    "created_at": date_.replace(tzinfo=None)
                }
                new_list.append(w_record)
            else:
                # update record
                date_ = arrow.get(str(create_date)).datetime

                if name != attribute_value.name:
                    w_record['name'] = name
                if display_name != attribute_value.display_name:
                    w_record['display_name'] = display_name
                if bool(ptav_active) != attribute_value.ptav_active:
                    w_record['ptav_active'] = bool(ptav_active)
                if int(product_attribute_value_id[0]) != attribute_value.product_attribute_value_id:
                    w_record['product_attribute_value_id'] = int(
                        product_attribute_value_id[0])
                if int(attribute_line_id[0]) != attribute_value.attribute_line_id:
                    w_record['attribute_line_id'] = int(attribute_line_id[0])
                if float(price_extra) != attribute_value.price_extra:
                    w_record['price_extra'] = float(price_extra)
                if list(exclude_for) != attribute_value.exclude_for:
                    w_record['exclude_for'] = list(
                        exclude_for)
                if int(product_tmpl_id[0]) != attribute_value.product_tmpl_id:
                    w_record['product_tmpl_id'] = int(product_tmpl_id[0])
                if int(attribute_id[0]) != attribute_value.attribute_id:
                    w_record['attribute_id'] = int(attribute_id[0])
                if list(ptav_product_variant_ids) != attribute_value.ptav_product_variant_ids:
                    w_record['ptav_product_variant_ids'] = list(
                        ptav_product_variant_ids)
                if bool(is_custom) != attribute_value.is_custom:
                    w_record['is_custom'] = bool(is_custom)
                if display_type != attribute_value.display_type:
                    w_record['display_type'] = display_type
                if date_.replace(tzinfo=None) != attribute_value.created_at:
                    w_record['created_at'] = date_.replace(tzinfo=None)

                if w_record:
                    w_record['odoo_id'] = attribute_value.odoo_id
                    update_list.append(w_record)

                redundant_ids.append(int(id))

        return [new_list, update_list, redundant_ids]
    except:
        exceptionRaise('insertOrUpdate')

# Bulk update
async def bulkUpdateQuery(session_, model_, pk_, values_):
    try:
        mappings = []
        for x in values_:
            print(x)
            stmt = update(model_).where(model_.odoo_id == x['odoo_id']).\
                where(model_.deleted_at.is_(None)).\
                values(x).\
                returning(model_.id)

            result = await session_.execute(stmt)
            mappings.append(result.scalar())

        return mappings
    except:
        exceptionRaise('bulkUpdateQuery')

async def cronAddUpdateProcess(session, body):
    if validate_list(body):
        body_records = body
    else:
        body_records = [body]
    output_list = []
    [new_list, update_list, redundant_ids] = await insertOrUpdate(session, body_records, True)
    if new_list:
        # Post new record
        result = await insertQuery(session, ProductAttributeValue, new_list)

        for u in result:
            output_list.append(u.id)

    if update_list:
        result = await bulkUpdateQuery(session, ProductAttributeValue, redundant_ids, update_list)

        for u in result:
            output_list.append(u)

    if (not output_list) and (not redundant_ids):
        await session.rollback()

    return output_list

# Cron auto feed to db func
async def migrateProdAttributeValueToDB(app):
    async with app.ctx.db.begin() as conn:
        # TODO: async func can await call from odoo, need improvements?
        output = await get_prod_attr_value()
        _ = await cronAddUpdateProcess(conn, output)

        await conn.commit()
        await conn.close()
    return True


import sys
from collections import deque

input = sys.stdin.readline
n, m, r = map(int, input().split())
graph = [[] for _ in range(n+1)]

for _ in range(m):
    a, b = map(int, input().split())
    graph[a].append(b)
    graph[b].append(a)

q = deque()
visited = [0] * (n+1)

q.append(r)
count = 1
visited[r] = count

while q:
    a = q.popleft()
    graph[a].sort()
    for i in graph[a]:
        if visited[i] == 0:
            count += 1
            visited[i] = count
            q.append(i)

for k in range(1, n+1):
    print(visited[k])


from pyspark.sql import functions as F
from pyspark.sql.types import *

from datetime import datetime

from dataquality_bnr.yamlHandler import verificationSuite as yVS

from pydeequ.repository import *
from pydeequ.verification import *

def addCheckConfigs(checkResult_df, checkParameters):
    df=checkResult_df

    checkLevel = checkParameters["level"]
    description = checkParameters["description"]

    df = df.withColumn("check", F.lit(description))
    df = df.withColumn("check_level",F.when(F.col("check_level")=="Success", F.col("check_level"))
                       .otherwise(F.lit(checkLevel)))
    df = df.withColumn("check_status",
                       F.when(F.col("check_status")=="Success", F.col("check_status"))
                       .otherwise(F.lit(checkLevel)))

    return df

def addDatasetDate(successMetrics_df, resultKeyParameters):
    df=successMetrics_df

    dataset_date = resultKeyParameters["dataset_date"]

    df=df.withColumn("dataset_date",F.lit(dataset_date))

    return df

def addKeyTags(successMetrics_df, resultKeyParameters):
    df=successMetrics_df

    key_tags = resultKeyParameters["key_tags"]

    tags=[]
    for k in key_tags:
        key = str(k)
        value = str(key_tags[k])
        tags.append(key+":"+value)

    df=df.withColumn("tags",F.array([F.lit(tag) for tag in tags]))

    return df

def addResultKey(successMetrics_df, resultKeyParameters):
    df=successMetrics_df

    df = addDatasetDate(df, resultKeyParameters)
    df = addKeyTags(df, resultKeyParameters)

    return df

def addJoinKey(checkResult_df):
    df = checkResult_df

    df = (df
          .withColumn("name",F.regexp_extract("constraint","(\(\w+\()", 1))
          .withColumn("instance",F.regexp_extract("constraint","(\(\w+\,)", 1))
          .withColumn("name",F.regexp_replace("name","\(", ""))
          .withColumn("instance",F.regexp_replace("instance","[(,]", ""))
          .withColumn("instance",F.when(F.col("instance")=="",F.lit("*")).otherwise(F.col("instance")))
         )
    return df

def addFormatedDate(successMetrics_df, dataset_date):
    df = successMetrics_df

    dt_obj = datetime.fromtimestamp(int(dataset_date/1000))
    dateYYMMDD = str(dt_obj.date())
    df = (df.withColumn("YY_MM_DD",F.lit(dateYYMMDD)))

    return df

def transformCheckResult(checkResult_df, checkParameters):
    checkResult_df = addCheckConfigs(checkResult_df, checkParameters)
    checkResult_df = addJoinKey(checkResult_df)

    return checkResult_df
def transformSuccesMetrics(successMetrics_df, resultKeyParameters):
    successMetrics_df = addResultKey(successMetrics_df, resultKeyParameters)
    successMetrics_df = addFormatedDate(successMetrics_df, resultKeyParameters["dataset_date"])

    return successMetrics_df

def getConsolidatedDataframe(successMetrics_df, checkResult_df, verificationSuiteConfigs):
    """
    :param successMetrics:successMetrics Dataframe
    :param checkResult:checkResult Dataframe
    :param verificationSuiteConfigs: dictionary containing verificationSuite configs
    """
    checkParameters=verificationSuiteConfigs["checkParameters"]
    resultKeyParameters=verificationSuiteConfigs["resultKeyParameters"]

    checkResult_df = transformCheckResult(checkResult_df, checkParameters)
    successMetrics_df = transformSuccesMetrics(successMetrics_df, resultKeyParameters)

    consolidatedOutput = successMetrics_df.join(checkResult_df, on=["instance", "name"])

    return consolidatedOutput

def getConsolidatedOutput2(spark, currResult, vsYamlPath):
    successMetrics_df = VerificationResult.successMetricsAsDataFrame(spark, currResult)
    checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, currResult)

    vsConfigs = yVS.getVerificationSuiteConfigs(vsYamlPath)
    vsConfigs["resultKeyParameters"]["dataset_date"] = ResultKey.current_milli_time()

    consolidatedOutput = getConsolidatedDataframe(successMetrics_df,
                                                       checkResult_df,
                                                       vsConfigs)
    return consolidatedOutput

def getSuccessMetrics(consolidatedOutput):
    metricColumns=["entity","instance","name","value","dataset_date","YY_MM_DD","tags"]
    successMetrics = consolidatedOutput.select(*[metricColumns])
    return successMetrics

def getCheckResult(consolidatedOutput):
    checkResultColumns=["check","check_level",
                    "check_status","constraint",
                    "constraint_status","constraint_message"]
    checkResult = consolidatedOutput.select(*[checkResultColumns])
    return checkResult

################################################################################################################
#                                  New Consolidate Approach
################################################################################################################

def addVsConfigs(df, vsConfigs, dfType):
    df = addDatasetDate(df, vsConfigs["resultKeyParameters"])
    df = addFormatedDate(df, vsConfigs["resultKeyParameters"]["dataset_date"])

    if dfType == "successMetrics":
        df = addKeyTags(df, vsConfigs["resultKeyParameters"])
    elif dfType == "checkResult":
        df = addCheckConfigs(df, vsConfigs["checkParameters"])

    return df

def setColumnsOrder(df, dfType):
    if dfType == "successMetrics":
        successColumns=['entity', 'instance', 'name', 'value', 'dataset_date', 'YY_MM_DD', 'tags']
        df = df.select(successColumns)
    elif dfType == "checkResult":
        checkColumns=['check','check_level','check_status','constraint',
                     'constraint_status','constraint_message','dataset_date','YY_MM_DD']
        df = df.select(checkColumns)

    return df

def addCheckStatus(successMetrics_df, checkResult_df):
    """
    Read checkResult_df and add information to successMetrics_df
    """
    checkStatus_value=""
    first_failure = checkResult_df.filter(F.col("constraint_status")=="Failure").first()
    if first_failure == None:
        checkStatus_value="Success"
    else:
        check_level = checkResult_df.select("check_level").first()[0]
        checkStatus_value= check_level

    df = successMetrics_df.withColumn("check_status", F.lit(checkStatus_value))

    return df

def getOuputDFs(spark, currResult, vsYamlPath):
    print("getOuputDFs...")
    successMetrics_df = VerificationResult.successMetricsAsDataFrame(spark, currResult)
    checkResult_df = VerificationResult.checkResultsAsDataFrame(spark, currResult)

    vsConfigs = yVS.getVerificationSuiteConfigs(vsYamlPath)
    vsConfigs["resultKeyParameters"]["dataset_date"] = ResultKey.current_milli_time()

    successMetrics_df = addVsConfigs(successMetrics_df, vsConfigs, "successMetrics")
    checkResult_df = addVsConfigs(checkResult_df, vsConfigs, "checkResult")

    successMetrics_df = addCheckStatus(successMetrics_df, checkResult_df)

    return successMetrics_df, checkResult_df


import pandas as pd
import re
import numpy as np
from timeit import default_timer as timer
from datetime import timedelta
from alive_progress import alive_bar

def get_tweets(query, limit=1_000_000_000, readable_csv=False, csv_name='tweets'):
    """
    Tasks
    -----
        Gets tweets from Twitter.

    Parameters
    ----------
    query: str
        The query to be searched on Twitter.
    limit: int (default=1000000000)
        The limit of tweets to be searched.
    readable_csv: bool (default=False)
        If True, saves the tweets as a csv file.
    csv_name: str (default='tweets.csv')
        The name of the csv file to be saved.
    Returns
    -------
    dataframe: pandas.DataFrame
        The dataframe containing the tweets.
    """

    import snscrape.modules.twitter as sntwitter

    tweets = []

    with alive_bar(limit, force_tty=True) as bar:

        if limit == 1000000000: print("Limit not defined. Progress bar may work untidy.")
        for i, t in enumerate(sntwitter.TwitterSearchScraper(query).get_items()):
            if i >= limit:
                break
            else:
                tweets.append(
                    [t.id, t.url, t.media, t.date.strftime("%d/%m/%Y, %H:%M:%S"), t.retweetCount, t.likeCount,
                     t.quoteCount, t.hashtags, t.content, t.lang, t.user.location, t.cashtags, t.conversationId,
                     t.coordinates, t.inReplyToTweetId, t.inReplyToUser, t.mentionedUsers, t.links, t.place,
                     t.quotedTweet, t.renderedContent, t.replyCount, t.retweetCount, t.retweetedTweet, t.source,
                     t.sourceLabel, t.sourceUrl, t.links, t.user, t.user.username,
                     t.user.created.strftime("%d-%m-%Y %H:%M:%S"), t.user.renderedDescription, t.user.descriptionLinks,
                     t.user.displayname, t.user.favouritesCount, t.user.followersCount, t.user.friendsCount, t.user.id,
                     t.user.label, t.user.linkTcourl, t.user.linkUrl, t.user.listedCount, t.user.location,
                     t.user.mediaCount, t.user.profileBannerUrl, t.user.profileImageUrl, t.user.protected,
                     t.user.rawDescription, t.user.statusesCount, t.user.url, t.user.username, t.user.verified])
            bar()

    dataframe = pd.DataFrame(tweets, columns=['id', 'url', 'media', 'date', 'retweet_count', 'like_count', 'quoteCount',
                                              'hashtags', 'content', 'lang', 'user_location',
                                              'cashtags', 'conversation_id', 'coordinates', 'inReplyToTweetId',
                                              'inReplyToUser', 'mentionedUsers', 'out_links', 'place',
                                              'quotedTweet', 'renderedContent', 'replyCount', 'retweetCount',
                                              'retweetedTweet', 'source', 'sourceLabel', 'sourceUrl', 'tco_out_links',
                                              'user', 'user_name', 'user_created', 'user_description',
                                              'user_descriptionUrls', 'user_display_name', 'user_favouritesCount',
                                              'user_followersCount', 'user_friendsCount', 'user_id', 'user_label',
                                              'user_link_Tco_url', 'user_linkUrl', 'user_listedCount', 'user_location',
                                              'user_media_count', 'user_profile_banner_url', 'user_profile_image_url',
                                              'user_protected', 'user_raw_description', 'user_statuses_count',
                                              'user_url', 'user_username', 'user_verified'])

    if readable_csv:
        dataframe.to_csv(csv_name + '.csv', index=True, encoding='utf-8-sig')
        dataframe.content.to_csv(csv_name + '_readable.csv', index=True, encoding='utf-8-sig')
        print("CSV file created")

    print(f"Dataframe has {dataframe.shape[0]} tweets")
    return dataframe

def preprocessing(series, remove_hashtag=False, remove_mentions=False, remove_links=False, remove_numbers=False,
                  remove_short_text=False, remove_stopwords=False, lowercase=False, remove_punctuation=False,
                  remove_rare_words=False, rare_limit=5):
    """
    Tasks
    -----
        Preprocesses the given series.

    Parameters
    ----------
    series: pandas.Series
        The series to be preprocessed.
    lowercase: bool
        If True, converts all characters to lowercase.
    remove_hashtag: bool
        If True, removes hashtags (#) from the series.
    remove_mentions: bool
        If True, removes mentions (@) from the series.
    remove_links: bool
        If True, removes links from the series.
    remove_numbers: bool
        If True, removes numbers from the series.
    remove_short_text: bool
        If True, removes short texts (shorter than the given value) from the series.
    remove_stopwords: bool
        If True, removes stopwords (gets stopwords from stopwords.txt) from the series.
    remove_punctuation:
        If True, removes punctuation from the series.
    remove_rare_words:
        If True, removes rare words (occurs less than the given value) from the series.
    rare_limit: int (default=5)
        The limit for removing rare words.

    Returns
    -------
    series: pandas.Series
        The preprocessed series.
    """
    if lowercase:
        print("Lowercasing...")
        start = timer()
        series = series.str.lower()
        print(f"Lowercasing is done in {timedelta(seconds=timer() - start)}")

    if remove_hashtag:
        print("Removing hashtags...")
        start = timer()
        series = series.str.replace(r'((#)[^\s]*)\b', '', regex=True)
        print(f"Removed in {timedelta(seconds=timer() - start)}")

    if remove_mentions:
        print("Removing mentions...")
        start = timer()
        series = series.str.replace(r'((@)[^\s]*)\b', '', regex=True)
        print(f"Removed in {timedelta(seconds=timer() - start)}")

    if remove_links:
        print("Removing links...")
        start = timer()
        series = series.str.replace(r'\n', '', regex=True)
        series = series.apply(lambda x: re.split('https:\/\/.*', str(x))[0])
        print(f"Removed in {timedelta(seconds=timer() - start)}")

    if remove_numbers:
        print("Removing numbers...")
        start = timer()
        series = series.str.replace(r'\d+', '', regex=True)
        print(f"Removed in {timedelta(seconds=timer() - start)}")

    if remove_short_text:
        print("Removing short texts...")
        start = timer()
        series = series.apply(lambda x: re.sub(r'\b\w{1,2}\b', '', x))
        print(f"Removed in {timedelta(seconds=timer() - start)}")

    if remove_stopwords:
        print("Removing stopwords...")
        start = timer()
        with open('assets/stopwords.txt', 'r', encoding='utf-8') as f:
            stop = [line.strip() for line in f]
        series = series.apply(lambda x: ' '.join([word for word in x.split() if word.lower() not in stop]))
        print(f"Stopwords are removed in {timedelta(seconds=timer() - start)}")

    if remove_punctuation:
        print("Removing punctuation...")
        start = timer()
        series = series.str.replace(r"((')[^\s]*)\b", '', regex=True)
        series = series.str.replace(r'[^\w\s]', '', regex=True)
        print(f"Removed in {timedelta(seconds=timer() - start)}")

    if remove_rare_words:
        print("Removing rare words...")
        start = timer()
        whole_count = pd.Series(" ".join(series).split()).value_counts()
        print(f"There are {whole_count.count()} words in the series")
        print(f"%{round(whole_count[whole_count <= rare_limit].count() / whole_count.count() * 100, 2)} of "
              f"words appear lesser than rare limit ({rare_limit}) times")
        to_remove = whole_count[whole_count <= rare_limit]
        print(f"Removing rare words...")
        series = series.apply(lambda x: " ".join(x for x in x.split() if x not in to_remove))
        print(f"{len(to_remove)} rare words removed")
        print(f"Removed in {timedelta(seconds=timer() - start)}")

    return series

def sentiment(series, model_name="savasy/bert-base-turkish-sentiment-cased"):
    """
    Tasks
    -----
        Calculates the sentiment of the given series.

    Parameters
    ----------
    series: pandas.Series
        The series to be preprocessed.
    model_name: str (default="savasy/bert-base-turkish-sentiment-cased")
        The name of the model to be used for sentiment analysis.

    Returns
    -------
    label: list
        The sentiment label of the given series.
    score: list
        The sentiment score of the given series.
    """

    from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline

    label, score = [], []
    model = AutoModelForSequenceClassification.from_pretrained(model_name)
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    sa = pipeline("sentiment-analysis", tokenizer=tokenizer, model=model)
    for i, k in enumerate(series):
        result = sa(k)
        label.append(result[0]['label'])
        score.append(result[0]['score'])
        if i % 100 == 0 and i != 0:
            print(f"Sentiment analysis of {i} tweets is done")
    return pd.DataFrame({'label': label, 'score': score})

def translator(series, target_language="en", from_language="auto", secure_translations=False, secure_frequency=1000):
    """
    Tasks
    -----
        Translate given series to target language. The process is too slow due to the API limit.
        if you want to secure your translations, you can use secure_translations=True. It will save the translations
        in every secure_frequency. Due to the API limit, it is recommended to use secure_translations=True.
        if API give you an error and secure_translations=False you will lose all the translations.
        Enable secure_translations may slow down the process.

    Parameters
    ----------
    series: pandas.Series
        The series to be translated.
    target_language: str (default="en")
        The target language to be translated.
    from_language: str (default="auto")
        The language of the given series. If "auto", the language will be detected automatically.
    secure_translations: bool (default=False)
        If True, the translation will be done in batches.
    secure_frequency: int (default=1000)
        The frequency of the batches.

    Returns
    -------
    series: pandas.Series
        The translated series.
    """

    from mtranslate import translate

    secure_list, translate_list = [], []
    start = timer()
    print("Translation started, It's too slow due to API limit.")
    print("100 tweets nearly takes 40 seconds")
    print("You will see the progress in the console soon.")
    for i, k in enumerate(series):
        try:
            translate_list.append(translate(k, target_language, from_language))
            if secure_translations:
                if i % secure_frequency == 0 and i != 0:
                    print(f"{i} tweets are translated and secured")
                    secure_list = translate_list.copy()
                elif i % 100 == 0 and i != 0:
                    print(f"{i} tweets are translated and {timedelta(seconds=(timer() - start))} time elapsed")
            elif i % 100 == 0 and i != 0:
                print(f"{i} tweets are translated and {timedelta(seconds=(timer() - start))} time elapsed")
        except Exception as e:
            print(e)
            return secure_list, True
    print(f"Translation is done in {timedelta(seconds=(timer() - start))}")
    return translate_list, False

def language_detect(series):
    """
    Tasks
    -----
        Detects the language of the given series.

    Parameters
    ----------
    series: pandas.Series
        The series to be detected.
    Returns
    -------
    series: pandas.Series
        The language of the given series.
    """

    from langdetect import detect

    detected = [detect(k) for k in series]
    print(pd.Series(detected).value_counts(normalize=True))
    return detected

def get_models(x, y, test_size=0.25, random_state=15, classification=False, average='binary', order_type='acc',
               info=False):
    """
    Tasks
    -----
        This functions returns scores of baseline models for classification and regression problems.
    Parameters
    ----------
    x: pandas.DataFrame
        The features of the dataset.
    y: pandas.Series
        The target of the dataset.
    test_size: float (default=0.2)
        The size of the test set.
    random_state: int (default=42)
        The random state of the train test split.
    classification
        If True, the function will work on classification and returns their score.
    average: str (default='binary')
        The average method of the classification report.
    order_type: str (default='acc')
        The order type of the scores. If 'acc', the scores will be ordered by accuracy. If 'f1', the scores will be
        ordered by f1 score. If 'precision', the scores will be ordered by precision score. If 'recall', the scores will
        be ordered by recall score. If 'time' scores will be ordered by time.
    info: bool (default=False)
        If True, the function will print while training.
    Returns
    -------
    print: str
        The scores of the baseline models.
    """

    from sklearn.metrics import accuracy_score, mean_squared_error, f1_score, precision_score, recall_score
    from sklearn.model_selection import train_test_split

    # Tum Base Modeller (Classification)
    from sklearn.linear_model import LogisticRegression
    from sklearn.neighbors import KNeighborsClassifier
    from sklearn.tree import DecisionTreeClassifier
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.ensemble import GradientBoostingClassifier
    from catboost import CatBoostClassifier
    from lightgbm import LGBMClassifier
    from sklearn.svm import SVC

    # Tum Base Modeller (Regression)
    from catboost import CatBoostRegressor
    from lightgbm import LGBMRegressor
    from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
    from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
    from sklearn.neighbors import KNeighborsRegressor
    from sklearn.svm import SVR
    from sklearn.tree import DecisionTreeRegressor
    from xgboost import XGBRegressor

    x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=test_size, random_state=random_state)

    all_models = []
    order_types = {'acc': 1, 'precision': 2, 'recall': 3, 'f1': 4, 'time': 5}
    if classification:
        models = [('LR', LogisticRegression(random_state=random_state)),
                  ('KNN', KNeighborsClassifier()),
                  ('CART', DecisionTreeClassifier(random_state=random_state)),
                  ('RF', RandomForestClassifier(random_state=random_state)),
                  ('SVM', SVC(gamma='auto', random_state=random_state)),
                  ('GradientBoosting', GradientBoostingClassifier(random_state=random_state)),
                  ("LightGBM", LGBMClassifier(random_state=random_state)),
                  ("CatBoost", CatBoostClassifier(verbose=False, random_state=random_state))]

        for name, model in models:
            start = timer()
            if info: print(f"{name} is training")
            model.fit(x_train, y_train)
            y_pred = model.predict(x_test)
            acc_test = accuracy_score(y_test, y_pred)
            precision = precision_score(y_test, y_pred, average=average)
            recall = recall_score(y_test, y_pred, average=average)
            f1 = f1_score(y_test, y_pred, average=average)
            values = dict(name=name, acc_test=acc_test, precision=precision, recall=recall, f1=f1,
                          train_time=str(timedelta(seconds=(timer() - start)))[-15:])
            if info: print(f"{name} is done in {timedelta(seconds=(timer() - start))}")
            all_models.append(values)
        sort_method = False

    else:
        models = [('LR', LinearRegression()),
                  ("Ridge", Ridge()),
                  ("Lasso", Lasso()),
                  ("ElasticNet", ElasticNet()),
                  ('KNN', KNeighborsRegressor()),
                  ('CART', DecisionTreeRegressor()),
                  ('RF', RandomForestRegressor()),
                  ('SVR', SVR()),
                  ('GBM', GradientBoostingRegressor()),
                  ("XGBoost", XGBRegressor()),
                  ("LightGBM", LGBMRegressor()),
                  ("CatBoost", CatBoostRegressor(verbose=False))]

        for name, model in models:
            print(f"{name} is training")
            model.fit(x_train, y_train)
            y_pred_test = model.predict(x_test)
            y_pred_train = model.predict(x_train)
            rmse_test = np.sqrt(mean_squared_error(y_test, y_pred_test))
            rmse_train = np.sqrt(mean_squared_error(y_train, y_pred_train))
            values = dict(name=name, RMSE_TRAIN=rmse_train, RMSE_TEST=rmse_test)
            all_models.append(values)
        sort_method = True

    all_models_df = pd.DataFrame(all_models)
    if order_type == 'time':
        sort_method = True
    all_models_df = all_models_df.sort_values(all_models_df.columns[order_types[order_type]], ascending=sort_method)
    print(f"\nAll models are done --- ordered by {order_type}")
    print(all_models_df.to_markdown())
    return None

def create_tfidf(series):
    """
    Tasks
    -----
        Creates tfidf matrix for the given series.

    Parameters
    ----------
    series: pandas.Series
        The series to be transformed.

    Returns
    -------
    df_tfidf_vect: pandas.DataFrame
        The tfidf matrix.
    """

    from sklearn.feature_extraction.text import TfidfVectorizer

    tfidf_vectorized = TfidfVectorizer(analyzer='word')
    tfidf_wm = tfidf_vectorized.fit_transform(series)
    tfidf_tokens = tfidf_vectorized.get_feature_names_out()
    df_tfidf_vect = pd.DataFrame(data=tfidf_wm.toarray(), columns=tfidf_tokens)

    return df_tfidf_vect

def describe_series(series):
    """
    Tasks
    -----
        Describes the given series.

    Parameters
    ----------
    series: pandas.Series
        The series to be described.

    Returns
    -------
    print: str
        The description of the given series.
    """
    most_repeated_word = pd.Series(" ".join(series).split()).value_counts().index[1]
    most_repeated_count = pd.Series(" ".join(series).split()).value_counts()[1]
    average_word_count = np.mean(series.apply(lambda x: len(x.split())))

    print(f"""
    Missing Values: {series.isnull().sum()}
    Series has {len(series)} rows.
    Series has {pd.Series(" ".join(series).split()).value_counts().count()} unique words.
    Average word count {round(average_word_count, 3)}
    Most repeated word is "{most_repeated_word}".
    "{most_repeated_word}" is repeated {most_repeated_count} times.
    """)
    return None

def tr_en_char_translate(series):
    """
    Tasks
    -----
        Convert Turkish characters to English characters.

    Parameters
    ----------
    series: pandas.Series
        The series to be translated.

    Returns
    -------
    series: pandas.Series
        The translated series.
    """
    series = series.str.replace('ı', 'i')
    series = series.str.replace('ü', 'u')
    series = series.str.replace('ö', 'o')
    series = series.str.replace('ğ', 'g')
    series = series.str.replace('ş', 's')
    series = series.str.replace('ç', 'c')
    return series


#!/usr/bin/env python
# coding=utf8
import json
import urllib
from model.setting import withBase, basecfg, baseorm, pack
from model.base import Task, Section, Flow, Article, Permit
from flask import Blueprint, request, Response, render_template, g
from .. import select, unselect

task = Blueprint('task', __name__, template_folder='template')

from monitor import *
from statistics import *

@task.route('/list', methods=['GET'])
@withBase(basecfg.R, resutype='DICT')
def tasklist():
    user = request.user
    skip = int(request.args.get('skip', 0))
    limit = int(request.args.get('limit', 10))
    keyword = request.args.get('keyword')

    condition = {'status':{'$gt':0}}
    if keyword:
        pack(Task, keyword, condition)
    total = Task.count(condition)
    tasks = Task.queryAll(user, condition, projection={'name':1, 'extra':1}, sort=[('update_time', -1)], skip=skip, limit=limit)
    result = {"appname":g.appname, "user":user, "task":tasks, 'total':total}
    result = json.dumps({'code':1, 'msg':'', 'res':result}, ensure_ascii=False, sort_keys=True, indent=4).encode('utf8')
    return result

@task.route('/detail', methods=['DELETE', 'POST'])
@task.route('/detail/<tid>', methods=['GET', 'POST', 'DELETE'])
@withBase(basecfg.W, resutype='DICT', autocommit=True)
def taskdetail(tid=None):
    user = request.user
    tid = baseorm.IdField.verify(tid) if tid else tid
    if request.method == 'GET':
        user = request.user
        if tid is None:
            tid = request.args.get('tid')
        if tid is None:
            task = {'_id':'', 'aid':'', 'sid':'', 'name':'', 'extra':'', 'type':'ONCE', 'period':0, 'fid':'', 'params':'', 'worknum':6, 'queuetype':'M', 'worktype':'THREAD', 'timeout':30, 'category':'', 'push_url':'', 'pull_url':'', 'tag':'', 'own':True}
        else:
            projection = {'aid':1, 'sid':1, 'name':1, 'extra':1, 'type':1, 'period':1, 'fid':1, 'params':1, 'worknum':1, 'queuetype':1, 'worktype':1, 'timeout':1, 'category':1, 'push_url':1, 'tag':1, 'creator':1}
            task = Task.queryOne(user, {'_id':tid}, projection=projection)

            projection = {'name':1}
            task['article'] = {
                'label':'article',
                'key':'aid',
                'val':task['aid'],
                'url':'task/article',
                'options':[{'text':one['name'], 'value':one['_id']} for one in Article.queryAll(user, {}, projection=projection, limit=None)]
            }

            projection = {'name':1}
            task['flow'] = {
                'label':'flow',
                'key':'fid',
                'val':task['fid'],
                'url':'task/flow',
                'options':[{'text':one['name'], 'value':one['_id']} for one in Flow.queryAll(user, {'aid':task['aid']}, projection=projection)]
            }

            projection = {'name':1}
            task['section'] = {
                'label':'section',
                'key':'sid',
                'val':task['sid'],
                'url':'task/section',
                'options':[{'text':one['name'], 'value':one['_id']} for one in Section.queryAll(user, {'fid':task['fid']}, projection=projection)]
            }

            task['own'] = str(task['creator']) == user['_id']
            task['pull_url'] = 'http://%s/gdc/api/data/%s' % (request.host, str(task['_id']))
            del task['creator']
        task['queuetype'] = 'M' # QUEUETYPE.get(task['queuetype'], '')
        task['queuetype_options'] = [{'text':'local', 'value':'P'},
                                    {'text':'beanstalkd', 'value':'B'},
                                    {'text':'redis', 'value':'R'},
                                    {'text':'mongo', 'value':'M'}]
        task['worktype'] = 'THREAD' # WORKTYPE.get(task['worktype'], '')
        task['worktype_options'] = [{'text':'多线程', 'value':'THREAD'},
                                    {'text':'协程', 'value':'GEVENT'}]
        task['type_options'] = [{'text':'临时任务', 'value':'ONCE'},
                                {'text':'周期任务', 'value':'FOREVER'}]
        author = {}
        # for one in Permit.queryAll({'otype':'Task', 'oid':tid}, projection={'cid':1, '_id':0}):
        #     author[str(one['cid'])] = ''
        task['author'] = urllib.quote(json.dumps(author).encode('utf8'))

        projection = {'_id':1, 'username':1}
        task['creators'] = [{'text':one['username'], 'value':one['_id']} for one in Creator.queryAll(user, {'_id':{'$ne':user['_id']}}, projection=projection, limit=None)]
        task['select_updators'] = ','.join([str(one['cid']) for one in Permit.queryAll({'cid':{'$ne':user['_id']}, 'creator':user['_id'], 'oid':tid, 'otype':'Task', 'authority':{'$in':[2,3,6,7,10,11,14,15]}}, projection={'cid':1}, limit=None)])
        task['select_queryers'] = ','.join([str(one['cid']) for one in Permit.queryAll({'cid':{'$ne':user['_id']}, 'creator':user['_id'], 'oid':tid, 'otype':'Task', 'authority':{'$mod':[2, 1]}}, projection={'cid':1}, limit=None)])

        result = {"appname":g.appname, "user":user, "task":task}
        result = json.dumps({'code':1, 'msg':'', 'res':result}, ensure_ascii=False, sort_keys=True, indent=4).encode('utf8')
        return result
    elif request.method == 'POST':
        user = request.user
        name = request.form.get('name')
        extra = request.form.get('extra')
        category = request.form.get('category')
        tag = request.form.get('tag')
        period = request.form.get('period')
        push_url = request.form.get('push_url')
        aid = request.form.get('aid')
        fid = request.form.get('fid')
        sid = request.form.get('sid')
        params = request.form.get('params')
        timeout = request.form.get('timeout', 30)
        worknum = request.form.get('worknum', 6)
        queuetype = request.form.get('queuetype', 'P')
        worktype = 'THREAD'

        select_updators = request.form.get('select_updators')
        unselect_updators = request.form.get('unselect_updators')
        select_queryers = request.form.get('select_queryers')
        unselect_queryers = request.form.get('unselect_queryers')

        aid = baseorm.IdField.verify(aid) if aid is not None else aid
        fid = baseorm.IdField.verify(fid) if fid is not None else fid
        sid = baseorm.IdField.verify(sid) if sid is not None else sid

        if request.form.get('type') == 'ONCE':
            period = 0
            queuetype = 'P'
        else:
            queuetype = 'R'
        section = Section.queryOne(user, {'_id':sid})
        result = {'stat':1, 'desc':'success', 'data':{}}
        if tid is None:
            task = Task(name=name,
                extra=extra,
                category=category,
                tag=tag,
                type=request.form.get('type'),
                period=period,
                push_url=push_url,
                aid=aid,
                fid=fid,
                sid=sid,
                params=params,
                timeout=timeout,
                worknum=worknum,
                state=0,
                status=1,
                queuetype=queuetype,
                worktype=worktype,
                creator=user['_id'],
                updator=user['_id'],
                create_time=datetime.datetime.now())
            task['_id'] = Task.insert(user, task)
        else:
            task = {'name':name,
                'extra':extra,
                'category':category,
                'tag':tag,
                'type':request.form.get('type'),
                'period':period,
                'push_url':push_url,
                'params':params,
                'timeout':timeout,
                'worknum':worknum,
                'queuetype':queuetype,
                'worktype':worktype,
                'updator':user['_id'],
                'update_time':datetime.datetime.now()
            }
            if aid is not None:
                task['aid'] = aid
            if fid is not None:
                task['fid'] = fid
            if sid is not None:
                task['sid'] = sid
            Task.update(user, {'_id':tid}, {'$set':task})
            task['_id'] = tid
        unselect(unselect_queryers, 'Task', tid, user['_id'])
        unselect(unselect_updators, 'Task', tid, user['_id'])
        select(select_updators, 'Task', tid, user['_id'], 'update')
        select(select_queryers, 'Task', tid, user['_id'], 'query')
        result = {"appname":g.appname, "user":user, "task":task}
        result = json.dumps({'code':1, 'msg':'', 'res':result}, ensure_ascii=False, sort_keys=True, indent=4).encode('utf8')
        return result
    elif request.method == 'DELETE':
        if tid:
            Task.update(user, {'_id':tid}, {'$set':{'status':0}})
        else:
            ids = request.form.get('ids')
            for tid in ids.split(','):
                tid = baseorm.IdField.verify(tid)
                Task.update(user, {'_id':tid}, {'$set':{'status':0}})
        result = {"appname":g.appname, "user":user, "task":{}}
        result = json.dumps({'code':1, 'msg':'Delete it successfully.', 'res':result}, ensure_ascii=False, sort_keys=True, indent=4).encode('utf8')
        return result
    else:
        pass

@task.route('/article', methods=['GET'])
@withBase(basecfg.R, resutype='DICT')
def taskarticle():
    user = request.user
    uid = request.args.get('uid')
    if uid is None:
        articles = Article.queryAll(user, {}, projection={'name':1})
    else:
        uid = baseorm.IdField.verify(uid)
        articles = Article.queryAll(user, {'uid':uid}, projection={'name':1})
    result = {"appname":g.appname, "user":user, "options":[{'text':one['name'], 'value':one['_id']} for one in articles]}
    result = json.dumps({'code':1, 'msg':'', 'res':result}, ensure_ascii=False, sort_keys=True, indent=4).encode('utf8')
    return result

@task.route('/flow', methods=['GET'])
@withBase(basecfg.R, resutype='DICT')
def taskflow():
    user = request.user
    aid = request.args.get('aid')
    if aid is None:
        flows = Flow.queryAll(user, {}, projection={'name':1})
    else:
        aid = baseorm.IdField.verify(aid)
        flows = Flow.queryAll(user, {'aid':aid}, projection={'name':1})
    result = {"appname":g.appname, "user":user, "options":[{'text':one['name'], 'value':one['_id']} for one in flows]}
    result = json.dumps({'code':1, 'msg':'', 'res':result}, ensure_ascii=False, sort_keys=True, indent=4).encode('utf8')
    return result

@task.route('/section', methods=['GET'])
@withBase(basecfg.R, resutype='DICT')
def tasksection():
    user = request.user
    fid = request.args.get('fid')
    if fid is None:
        sections = Section.queryAll(user, {}, projection={'name':1})
    else:
        fid = baseorm.IdField.verify(fid)
        sections = Section.queryAll(user, {'fid':fid}, projection={'name':1})
    result = {"appname":g.appname, "user":user, "options":[{'text':one['name'], 'value':one['_id']} for one in sections]}
    result = json.dumps({'code':1, 'msg':'', 'res':result}, ensure_ascii=False, sort_keys=True, indent=4).encode('utf8')
    return result


import argparse
import os
import pickle
import click

from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error

import mlflow

EXPERIMENT_NAME = "random-forest-experiment"
mlflow.set_experiment(EXPERIMENT_NAME) #create experiment
mlflow.sklearn.autolog() #enable autolog

def load_pickle(filename: str):
    with open(filename, "rb") as f_in:
        return pickle.load(f_in)

@click.command()
@click.option(
    "--data_path",
    default="./output",
    help="Location where the processed NYC taxi trip data was saved"
)

def run_train(data_path: str):

    X_train, y_train = load_pickle(os.path.join(data_path, "train.pkl"))
    X_val, y_val = load_pickle(os.path.join(data_path, "val.pkl"))

    with mlflow.start_run(): # required to track each run of the experiment

        rf = RandomForestRegressor(max_depth=10, random_state=0)
        rf.fit(X_train, y_train)
        #mlflow autolog will capture info about training set auomatically. but we have to manually track it for validation and test sets:

        y_pred = rf.predict(X_val)

        rmse = mean_squared_error(y_val, y_pred, squared=False)
        mlflow.log_metric("rmse", rmse) #track validation rmse

if __name__ == '__main__':
    run_train()


__copyright__ = "Copyright 2017 Birkbeck, University of London"
__author__ = "Martin Paul Eve & Andy Byers"
__license__ = "AGPL v3"
__maintainer__ = "Birkbeck Centre for Technology and Publishing"

from django.contrib import admin
from proofing import models

class ProofingAssignmentAdmin(admin.ModelAdmin):
    list_display = ('pk', 'article', 'proofing_manager', 'editor', 'assigned', 'completed')
    list_filter = ('proofing_manager', 'editor')
    raw_id_fields = ('article', 'proofing_manager', 'editor')

class ProofingRound(admin.ModelAdmin):
    list_display = ('pk', 'assignment', 'number', 'date_started')
    list_filter = ('assignment',)
    raw_id_fields = ('assignment',)

class ProofingTaskAdmin(admin.ModelAdmin):
    list_display = ('pk', 'round', 'proofreader', 'assigned', 'due', 'accepted', 'completed', 'cancelled')
    list_filter = ('round', 'proofreader')
    raw_id_fields = ('round', 'proofreader')
    filter_horizontal = ('galleys_for_proofing', 'notes')

class CorrectionTaskAdmin(admin.ModelAdmin):
    list_display = ('pk', 'proofing_task', 'typesetter', 'assigned', 'due', 'accepted', 'completed')
    list_filter = ('proofing_task', 'typesetter')
    raw_id_fields = ('proofing_task', 'typesetter')
    filter_horizontal = ('galleys',)

class NoteAdmin(admin.ModelAdmin):
    list_display = ('pk', 'galley', 'creator', 'date_time')
    list_filter = ('galley', 'creator')
    search_fields = ('text',)
    raw_id_fields = ('galley', 'creator')

admin_list = [
    (models.ProofingAssignment, ProofingAssignmentAdmin),
    (models.ProofingRound, ProofingRound),
    (models.ProofingTask, ProofingTaskAdmin),
    (models.TypesetterProofingTask, CorrectionTaskAdmin),
    (models.Note, NoteAdmin),
]

[admin.site.register(*t) for t in admin_list]


#reads the genome, cuts on TTAA, size selects, then outputs a .txt file of all the fragments
#usage - cut.py genomein txtout minsize maxmize
#example cut.py parent2_ref.fa outp2.txt 100 350
import sys
from Bio import SeqIO

def cutme(seq,seqid,mindist,maxdist,outputfile):
	counter=-1
	filetowrite=open(outputfile, 'a')
	while counter<len(seq)-3:
		count2=seq.find("TTAA",counter+1)
		if count2==-1:
			break
		if count2-counter > mindist and count2-counter < maxdist:
			if counter==-1:
				countprint=0
			else:
				countprint=counter
			print(">",str(seqid),str(counter),str(count2), file=filetowrite,sep='_')
			print(seq[countprint:count2+4],file=filetowrite)
		counter=count2
	filetowrite.close()

handle = open(sys.argv[1], "rU")
records = list(SeqIO.parse(handle, "fasta"))
handle.close()
for i in records:
    i.seq=i.seq.upper()
    cutme(i.seq,i.id,int(sys.argv[3]),int(sys.argv[4]),sys.argv[2])


import sys
from collections import deque
n, m = map(int, sys.stdin.readline().split())
visited = [False]*(n+1)
net = [[] for _ in range(n+1)]

for _ in range(m):
    u, v = map(int, sys.stdin.readline().split())
    net[u].append(v)
    net[v].append(u)

def bfs(idx):
    q = deque()
    q.append(idx)
    visited[idx] = True
    while q:
        now = q.popleft()
        for i in range(len(net[now])):
            nxt = net[now][i]

            if visited[nxt] is False:
                q.append(nxt)
                visited[nxt]=True

cnt =0
for i in range(1, n+1):
    if visited[i] is False:
        bfs(i)
        cnt+=1

print(cnt)

# import sys
# sys.setrecursionlimit(10000)
# n, m = map(int, sys.stdin.readline().split())
# visited = [False]*(n+1)
# net = [[] for _ in range(n+1)]
# for _ in range(m):
#     u, v = map(int, sys.stdin.readline().split())
#     net[u].append(v)
#     net[v].append(u)
#
# def dfs(idx):
#     visited[idx] = True
#     for i in net[idx]:
#         if visited[i] is False:
#             dfs(i)
#
# cnt =0
# for i in range(1, n+1):
#     if visited[i] is False:
#         dfs(i)
#         cnt+=1
#
# print(cnt)


import tkinter as tk
from pytube import YouTube
from tkinter import messagebox
import pyperclip
import threading

def paste_text():
    paste = pyperclip.paste()
    user_link.insert(tk.END,paste)

def download_video():
    #import urf from entry
    url = user_link.get()

    if not url :
        messagebox.showerror("Error","URL cannot be empty")
        return

    #eksekusi download
    try:
        ytb = YouTube(url)
        #resolusi youtube
        stream = ytb.streams.get_highest_resolution()

        def download_withLoading():
            eksekusi.config(state=tk.DISABLED)
            loading_label.config(text="Dowloading...",font=("Helvetica",12))
            for _ in stream.download(output_path="download"):
                pass
            messagebox.showinfo("Download Complete", "Video downloaded successfully.")
            loading_label.config(text="")
            eksekusi.config(state=tk.NORMAL)

        download_execution = threading.Thread(target=download_withLoading)
        download_execution.start()

    except Exception as e :
        messagebox.showerror("Error",f"An eror occurred: {str(e)}")

#jendela utama dari tkinter
root = tk.Tk()
root.title("youtube downloader")
root.geometry("250x170")
root.resizable(False,False)

#isi dari jendela tkinter
label1 = tk.Label(root,text="gift me your link youtube")
label1.pack()

user_link = tk.Entry(root)
user_link.pack()

title_video = tk.Entry(root)
title_video.pack
button_paste = tk.Button(root,text="paste",command=paste_text)
button_paste.pack(padx=5,pady=5)

#button eksekusi
eksekusi = tk.Button(root,text="download",command=download_video)
eksekusi.pack(padx=7,pady=5)
loading_label = tk.Label(root,text="",font=("Helvetica", 12))
loading_label.pack()

root.mainloop()


from roboflow import Roboflow
from PIL import Image
import cv2
import os, glob
import numpy as np

rf = Roboflow(api_key="GET_ROBOFLOW_API_KEY")
project = rf.workspace().project("car-models-nured")
model = project.version(2).model

rf_color = Roboflow(api_key="GET_ROBOFLOW_API_KEY")
project_color = rf_color.workspace().project("car-colors-1smyc")
model_color = project_color.version(5).model

font = cv2.FONT_HERSHEY_SIMPLEX
font_color = (255, 255, 255)
font_thickness = 3
font_scale = 2

box_color = (255, 255, 255)
box_thickness = 3
box_scale = 2

distance_color = (255, 255, 255)
distance_thickness = 3

# loop through folder of images
file_path = "test_images/" # folder of images to test - saved to output
# file_path = "images/single_image" # single image - saved to single_output
extention = ".jpg"
globbed_files = sorted(glob.glob(file_path + '*' + extention))
print(globbed_files)

counter = 0

for image_path in globbed_files:

    image_clean = cv2.imread(image_path)
    blk = np.zeros(image_clean.shape, np.uint8)

    pixel_ratio_array = []

    # infer on a local image
    response_json = model.predict(image_path, confidence=30, overlap=30).json()
    # print(response_json)

    for object in response_json["predictions"]:

        object_class = str(object['class'])
        object_class_text_size = cv2.getTextSize(object_class, font, font_scale, font_thickness)
        print("Class: " + object_class)
        object_confidence = str(round(object['confidence']*100 , 2)) + "%"
        print("Confidence: " + object_confidence)

        # pull bbox coordinate points
        x0 = object['x'] - object['width'] / 2
        y0 = object['y'] - object['height'] / 2
        x1 = object['x'] + object['width'] / 2
        y1 = object['y'] + object['height'] / 2
        box = (x0, y0, x1, y1)
        # print("Bounding Box Cordinates:" + str(box))

        image_cropped = image_clean[int(y0):int(y1), int(x0):int(x1)]

        image_cropped_path = "cropped/cropped_" + object_class + "_" + str(counter) + ".jpg"

        cv2.imwrite(image_cropped_path, image_cropped)

        # infer on a local image
        response_color_json = model_color.predict(image_cropped_path).json()

        object_color = response_color_json['predictions'][0]['top']

        print(object_color)

        if object_color == "Blue":
            box_color = (255, 0, 0)
            font_color = (255, 0, 0)
        elif object_color == "Grey":
            box_color = (127, 127, 127)
            font_color = (127, 127, 127)
        elif object_color == "Black":
            box_color = (0, 0, 0)
            font_color = (0, 0, 0)
        elif object_color == "Red":
            box_color = (0, 0, 255)
            font_color = (0, 0, 255)
        elif object_color == "Silver":
            box_color = (127, 127, 127)
            font_color = (127, 127, 127)
        elif object_color == "Green":
            box_color = (0, 180, 0)
            font_color = (0, 180, 0)
        elif object_color == "White":
            box_color = (255, 255, 255)
            font_color = (255, 255, 255)

        box_start_point = (int(x0), int(y0))
        class_font_start_point = (int(x0), int(y0)-10)
        confidence_font_start_point = (int(x0)+object_class_text_size[0][0], int(y0)-10)
        box_end_point = (int(x1), int(y1))

        # print(start_point)
        # print(end_point)

        image_drawn = cv2.rectangle(image_clean, box_start_point, box_end_point, box_color, box_thickness)

        image_drawn_blk = cv2.rectangle(blk, box_start_point, box_end_point, box_color, box_thickness)

        image_drawn = cv2.putText(image_drawn, object_class, class_font_start_point, font, font_scale, font_color, font_thickness, cv2.LINE_AA)

        image_drawn = cv2.putText(image_drawn, " - " + object_confidence, confidence_font_start_point, font, font_scale, font_color, font_thickness, cv2.LINE_AA)

        if object_class == "Hyundai Elantra":

            elantra_inches = 179.1

            elantra_x_min = int(x0)
            elantra_x_max = int(x1)
            elantra_y_min = int(y0)
            elantra_y_max = int(y1)

            elantra_width = object['width']
            elantra_half_height = int(object['height'] / 2)

            pixel_to_inches = elantra_width / elantra_inches
            pixel_ratio_array.append(pixel_to_inches)
            # print(pixel_to_inches)

        if object_class == "Kia Rio":

            rio_inches = 160

            rio_x_min = int(x0)
            rio_x_max = int(x1)
            rio_y_min = int(y0)
            rio_y_max = int(y1)

            rio_width = object['width']
            rio_half_height = int(object['height'] / 2)

            pixel_to_inches = rio_width / rio_inches
            pixel_ratio_array.append(pixel_to_inches)
            # print(pixel_to_inches)

    cv2.imwrite("boxes" + str(counter) + ".jpg", image_drawn)

    counter += 1

    try:
        if elantra_width and rio_width:

            print(np.mean(pixel_ratio_array))
            pixel_to_inches = int(np.mean(pixel_ratio_array))
            font_color = (255, 255, 255)

            x_calc_1 = abs(elantra_x_min - rio_x_min)
            x_calc_2 = abs(elantra_x_min - rio_x_max)
            x_calc_3 = abs(elantra_x_max - rio_x_max)
            x_calc_4 = abs(elantra_x_max - rio_x_min)

            # print(str(x_calc_1))
            # print(str(x_calc_2))
            # print(str(x_calc_3))
            # print(str(x_calc_4))

            min_pixel_distance = min(x_calc_1, x_calc_2, x_calc_3, x_calc_4)

            if x_calc_1 == min_pixel_distance:

                estimated_distance_inches = round(min_pixel_distance / pixel_to_inches, 2)
                distance_start_point = (rio_x_min, rio_y_max - rio_half_height)
                distance_end_point = (elantra_x_min, elantra_y_max - elantra_half_height)
                distance_text_point = (rio_x_min + int(x_calc_1 / 5), (elantra_y_max - elantra_half_height) - 10)

                print("x_calc_1 - smallest: " + str(x_calc_1))
                print("Estimated Inches: " + str(estimated_distance_inches))
                # print(distance_start_point)
                # print(distance_end_point)

            elif x_calc_2 == min_pixel_distance:

                estimated_distance_inches = round(min_pixel_distance / pixel_to_inches, 2)
                distance_start_point = (rio_x_max, rio_y_max - rio_half_height)
                distance_end_point = (elantra_x_min, elantra_y_max - elantra_half_height)
                distance_text_point = (rio_x_max + int(x_calc_2 / 5), (elantra_y_max - elantra_half_height) - 20)

                print("x_calc_2 - smallest: " + str(x_calc_2))
                print("Estimated Inches: " + str(estimated_distance_inches))
                # print(distance_start_point)
                # print(distance_end_point)

            elif x_calc_3 == min_pixel_distance:

                estimated_distance_inches = round(min_pixel_distance / pixel_to_inches, 2)
                distance_start_point = (elantra_x_max, elantra_y_max - elantra_half_height)
                distance_end_point = (rio_x_max, rio_y_max - rio_half_height)
                distance_text_point = (elantra_x_max + int(x_calc_3 / 5), (elantra_y_max - elantra_half_height) - 20)

                print("x_calc_3 - smallest: " + str(x_calc_3))
                print("Estimated Inches: " + str(estimated_distance_inches))
                # print(distance_start_point)
                # print(distance_end_point)

            else:

                estimated_distance_inches = round(min_pixel_distance / pixel_to_inches, 2)
                distance_start_point = (elantra_x_max, elantra_y_max - elantra_half_height)
                distance_end_point = (rio_x_min, rio_y_max - rio_half_height)
                distance_text_point = (elantra_x_max + int(x_calc_4 / 5), (elantra_y_max - elantra_half_height) - 20)

                print("x_calc_4 - smallest: " + str(x_calc_4))
                print("Estimated Inches: " + str(estimated_distance_inches))
                # print(distance_start_point)
                # print(distance_end_point)

            image_drawn = cv2.line(image_drawn, distance_start_point, distance_end_point, distance_color, distance_thickness)

            image_drawn = cv2.putText(image_drawn, "~" + str(estimated_distance_inches) + "in.", distance_text_point, font, font_scale, font_color, font_thickness, cv2.LINE_AA)

            cv2.imwrite("distance_estimate_" + str(counter) + ".jpg", image_drawn)

            # cv2.imwrite("distance_estimate_blk_" + str(counter) + ".jpg", image_drawn_blk)

            # cv2.imshow("distance_estimate_" + str(counter), image_drawn)
    except:
        pass

cv2.waitKey(0)