# Databricks notebook source
# MAGIC %md-sandbox
# MAGIC
# MAGIC

# MAGIC
# MAGIC

# COMMAND ----------

# MAGIC %md # Linear Regression
# MAGIC
# MAGIC Before you attempt to throw a neural network at a problem, you want to establish a __baseline model__. Often, this will be a simple model, such as linear regression. Once we establish a baseline, then we can get started with Deep Learning.
# MAGIC
# MAGIC The slides for the course can be found here.
# MAGIC
# MAGIC ## ![Spark Logo Tiny](https://files.training.databricks.com/images/105/logo_spark_tiny.png) In this lesson you will:

# MAGIC  - Build a linear regression model using scikit-learn and reimplement it in Keras
# MAGIC  - Modify the # of epochs
# MAGIC  - Visualize the loss during training

# COMMAND ----------

# MAGIC %run ./Includes/Classroom-Setup

# COMMAND ----------

# MAGIC %md
# MAGIC Let's start by making a simple array of features, and the label we are trying to predict is y = 2*X + 1.

# COMMAND ----------

import numpy as np
np.set_printoptions(suppress=True)

X = np.arange(-10, 11).reshape((21,1))
y = 2*X + 1

list(zip(X, y))

# COMMAND ----------

import matplotlib.pyplot as plt

plt.plot(X, y, "ro", label="True y")

plt.title("X vs. y")
plt.xlabel("X")
plt.ylabel("y")
plt.legend()

plt.show()

# COMMAND ----------

# MAGIC %md
# MAGIC Use sklearn to establish our baseline (for simplicity, we are using the same dataset for training and testing in this toy example, but we will change that later).
# MAGIC
# MAGIC NOTE: We are using the [Databricks ML Runtime](https://docs.databricks.com/runtime/mlruntime.html) which comes with [MLflow](https://mlflow.org/) pre-installed, as well as enabled MLflow autologging to automatically track our machine learning experiments. We'll do a deep dive into MLflow later, but you will see the output of the MLflow Run logged below.

# COMMAND ----------

from sklearn.linear_model import LinearRegression

lr = LinearRegression()

lr.fit(X, y)

# COMMAND ----------

# MAGIC %md
# MAGIC To get a sense of how well this model performed, let's import a metric of error estimation, the [mean_squared_error](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-error) in this case.

# COMMAND ----------

from sklearn.metrics import mean_squared_error

y_pred = lr.predict(X)
mse = mean_squared_error(y, y_pred)
print(mse)

# COMMAND ----------

# MAGIC %md
# MAGIC ### Visualize Predictions
# MAGIC
# MAGIC The mean squared error term above is one way to understand how well this model performs. But we can also use the model to predict values and see how well it matches the truth with a graph.

# COMMAND ----------

plt.plot(X, y, "ro", label="True y")
plt.plot(X, y_pred, label="Pred. y")

plt.title("X vs. True y and Pred. y (Linear Regression)")
plt.xlabel("X")
plt.ylabel("y")
plt.legend()

plt.show()

# COMMAND ----------

# MAGIC %md-sandbox
# MAGIC ## Keras
# MAGIC
# MAGIC Now that we have established a baseline model, let's see if we can build a fully-connected neural network that can meet or exceed our linear regression model. A fully-connected neural network is simply a set of matrix multiplications followed by some non-linear function (to be discussed later).
# MAGIC
# MAGIC Keras is a high-level API to build neural networks and was released by François Chollet in 2015. It is now the official high-level API of TensorFlow.
# MAGIC
# MAGIC ##### Steps to build a Keras model
# MAGIC

# COMMAND ----------

# MAGIC %md # 1. Define an n-Layer Neural Network
# MAGIC
# MAGIC Here, we need to specify the dimensions of our input and output layers. When we say something is an n-layer neural network, we count all of the layers except the input layer.
# MAGIC
# MAGIC A special case of neural network with no hidden layers and no non-linearities is actually just linear regression :).
# MAGIC
# MAGIC For the next few labs, we will use the Sequential model from Keras.

# COMMAND ----------

import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
tf.random.set_seed(42)

# The Sequential model is a linear stack of layers.
model = Sequential()

model.add(Dense(units=1, input_dim=1, activation="linear"))

# COMMAND ----------

# MAGIC %md
# MAGIC We can check the model definition by calling **`.summary()`**. Note the two parameters - any thoughts on why there are TWO?

# COMMAND ----------

model.summary()

# COMMAND ----------

# MAGIC %md # 2. Compile a Network
# MAGIC
# MAGIC To compile the network, we need to specify the loss function and which optimizer to use. We'll talk more about optimizers and loss metrics in the next lab.
# MAGIC
# MAGIC For right now, we will use **`mse`** (mean squared error) for our loss function, and the **`adam`** optimizer.

# COMMAND ----------

model.compile(loss="mse", optimizer="adam")

# COMMAND ----------

# MAGIC %md # 3. Fit a Network
# MAGIC
# MAGIC Let's fit our model on X and y, this will use the optimizer and loss function we set previously to train on the data.

# COMMAND ----------

model.fit(X, y)

# COMMAND ----------

# MAGIC %md
# MAGIC And take a look at the predictions.

# COMMAND ----------

keras_pred = model.predict(X)
keras_pred

# COMMAND ----------

# MAGIC %md
# MAGIC We can also see this model's **`mse`** using the same function as before from sklearn. Take note of how this compares to our **baseline model**.

# COMMAND ----------

mse = mean_squared_error(y, keras_pred)
print(mse)

# COMMAND ----------

# MAGIC %md
# MAGIC As before, we can use a plot to get a visual representation of how well this new model fits to the truth.

# COMMAND ----------

def keras_pred_plot(keras_pred):
    plt.clf()
    plt.plot(X, y, "ro", label="True y")
    plt.plot(X, keras_pred, label="Pred y")

    plt.title("X vs. True y and Pred. y (Keras)")
    plt.xlabel("X")
    plt.ylabel("y")
    plt.legend(numpoints=1)
    plt.show()

keras_pred_plot(keras_pred)

# COMMAND ----------

# MAGIC %md
# MAGIC What went wrong?? Turns out there a few more hyperparameters we need to set. Let's take a look at Keras documentation.
# MAGIC
# MAGIC The parameter **`epochs`** specifies how many passes you want over your entire dataset. Let's increase the number of epochs, and look at how the **`mse`** decreases.
# MAGIC
# MAGIC Here we are capturing the output of model.fit() as it returns a History object, which keeps a record of training loss values and metrics values at successive epochs, as well as validation loss values and validation metrics values (if applicable).

# COMMAND ----------

history = model.fit(X, y, epochs=20)

# COMMAND ----------

def view_model_loss():
    plt.clf()
    plt.plot(history.history["loss"])
    plt.title("Model Loss")
    plt.ylabel("Loss")
    plt.xlabel("Epoch")
    plt.show()

view_model_loss()

# COMMAND ----------

# MAGIC %md
# MAGIC This value for *`mse`* is still far higher than for our **baseline model**, so let's try increasing the epochs even more.

# COMMAND ----------

history = model.fit(X, y, epochs=4000)

# COMMAND ----------

# MAGIC %md
# MAGIC Let's inspect how our model decreased the loss (**`mse`**) as the number of epochs increases.

# COMMAND ----------

view_model_loss()

# COMMAND ----------

# MAGIC %md
# MAGIC Let's extract model weights and compare them to our equation for 2X+1.

# COMMAND ----------

print(model.get_weights())
predicted_w = model.get_weights()[0][0][0]
predicted_b = model.get_weights()[1][0]

print(f"predicted_w: {predicted_w}")
print(f"predicted_b: {predicted_b}")

# COMMAND ----------

# MAGIC %md
# MAGIC Wahoo! We were able to approximate y=2*X + 1 quite well! If we trained for some more epochs, we should be able to approximate this function exactly (at risk of overfitting of course).
# MAGIC
# MAGIC As a final note, let's see how the graph looks on this new set of weights.

# COMMAND ----------

# MAGIC %md # 4. Evaluate Network
# MAGIC
# MAGIC As mentioned previously, we want to make sure our neural network can beat our benchmark.

# COMMAND ----------

model.evaluate(X, y) # Prints loss value & metrics values for the model in test mode (both are MSE here)

# COMMAND ----------

# MAGIC %md # 5. Make Predictions

# COMMAND ----------

keras_pred = model.predict(X)
keras_pred

# COMMAND ----------

keras_pred_plot(keras_pred)

# COMMAND ----------

# MAGIC %md
# MAGIC Alright, this was a very simple, contrived example. Let's go ahead and make this a bit more interesting in the next lab!

# COMMAND ----------

# MAGIC %md
# MAGIC
# MAGIC ## Classroom Cleanup
# MAGIC
# MAGIC Run the following cell to remove lessons-specific assets created during this lesson:

# COMMAND ----------

DA.cleanup()

# COMMAND ----------

# MAGIC %md-sandbox
# MAGIC © 2022 Databricks, Inc. All rights reserved.

# MAGIC Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.

# MAGIC

# MAGIC Privacy Policy | Terms of Use | Support

DB/Temp/ml/deep-learning-with-databricks/Solutions/DL 01 - Linear Regression.py


import sys
from srcs.gameManager import GameManager
from srcs.commands.start import Start
from srcs.commands.end import End
from srcs.commands.turn import Turn
from srcs.commands.begin import Begin
from srcs.commands.info import Info
from srcs.commands.about import About
from srcs.commands.board import Board

# from rich import inspect, print

COMMANDS = ["START", "TURN", "BEGIN", "BOARD", "INFO", "END", "ABOUT"]
FUNCTIONS_STR = [
    "startCmd",
    "turnCmd",
    "beginCmd",
    "boardCmd",
    "infoCmd",
    "endCmd",
    "aboutCmd",
]

class Game:
    def __init__(self):
        self._commands = {
            "START": Start,
            "END": End,
            "TURN": Turn,
            "BEGIN": Begin,
            "INFO": Info,
            'ABOUT': About,
            'BOARD': Board,
        }
        self._gameManager = GameManager()

    def loop(self):
        while 1:
            lineinput = sys.stdin.readline()
            InputsCommands = lineinput.strip().split(" ")
            if InputsCommands[0] in COMMANDS:
                self._commands[InputsCommands[0]](InputsCommands, self._gameManager)
            else:
                print('ERROR unknown command "{}"'.format(InputsCommands[0]))

if __name__ == "__main__":
    foo = Game()
    foo.loop()


from struct import *

def ushort_uint(bBytes):
	x, y = unpack(">HI", bBytes)
	return (x, y)

def buf2latin(bBytes):
	l = unpack(">H", bBytes[:2])[0]
	s = str(l) + 's'
	return (l, unpack(s, bBytes[2:])[0].decode("latin_1"))

def ascii2buf(*vars):
	array = bytearray()
	size = pack(">L", len(vars))
	array += size
	for var in vars:
		size = pack(">H", len(var))
		array += size
		array += var.encode("ascii")
	return array


import os, subprocess
import scipy.io.wavfile as wav

# Script for listing the sample rate of files

path = 'Sounds'
vowels = os.listdir(path)

for vowel in vowels:
    (rate, signal) = wav.read("Sounds/{!s}".format(vowel))
    print(rate, vowel.replace('.wav', ''))

print()
RESAMPLE = input("Would you like to resample? (Y/N)  ")

if RESAMPLE == 'y' or 'Y':
    NEW_SAMPLE_RATE = int(input("Enter new sample rate in Hz. (Int)  "))

    for sound in vowels:
        with open('resample.praat', 'a') as f:
            f.write('Read from file: "{}/{}"\n'.format(path, sound))
            f.write('Resample: {!s}, 50\n'.format(NEW_SAMPLE_RATE))
            f.write('selectObject: "Sound {}_{!s}"\n'.format(sound.replace('.wav', ''), NEW_SAMPLE_RATE))
            f.write('Save as WAV file: "{}/{}_{!s}.wav"\n'.format(path, sound.replace('.wav', ''), NEW_SAMPLE_RATE))

subprocess.run(['./Praat/praat', '--run', './resample.praat'])


import os

from flask_bombril.r import R
from flask_bombril.utils import raise_with_stop

class UniqueFilename(object):
    def __init__(self, folder_full_path, message=R.string.unique_filename, stop=True):
        self.message = message
        self.folder_full_path = folder_full_path
        self.stop = stop

    def __call__(self, form, field):
        if callable(self.message):
            self.message = self.message()

        is_file = os.path.isfile(os.path.join(self.folder_full_path, field.data.filename))
        if is_file:
            raise_with_stop(self)


from main import *
from flask import Blueprint, render_template
from flask import request, redirect
from main.service import openAI_input, key, local_IO, tistory_post
import datetime

## 선언부 ===================================================================================
var_blue = Blueprint("blue", __name__, template_folder="templates", url_prefix='/blue')

## index페이지 호출 ===================================================================================
@var_blue.route('/', methods=['POST'])
def root_prompt():
    prompt = request.form['prompt']

    # 블로그 제목과 글을 생성합니다.
    content = openAI_input.prompt_factory.get_blog_content(key.openai_api_key, prompt)

    # 블로그 글의 요약문을 생성합니다.
    summary = openAI_input.prompt_factory.get_summary(key.openai_api_key, content)

    # Unsplash에서 이미지 URL을 가져옵니다.
    image_url = openAI_input.prompt_factory.get_image_url(key.unsplash_api_key, summary)

    # 생성된 글을 토대로 tag를 생성합니다.
    tags = openAI_input.prompt_factory.get_tag(key.openai_api_key, summary)

    #today
    today_now = datetime.datetime.now()
    today = today_now.strftime('%Y-%m-%d')

    # Save the data as a CSV file and IMG file.
    data = [prompt, tags, image_url, summary, content]
    local_IO.save_to_csv(data, prompt)
    local_IO.save_url_img(image_url, prompt)

    return render_template('index.html', prompt=prompt, content=content, image_url=image_url, today=today, summary=summary, tags=tags)

## modify페이지 호출 ===================================================================================
@var_blue.route('/modify', methods=['POST'])
def modify_content():

    prompt = request.form['prompt']
    dummy_data = local_IO.read_csv(prompt)

    tags = dummy_data['tags']
    image_url = dummy_data['image_path']
    summary = dummy_data['summary']
    content = dummy_data['content']

    #today
    today_now = datetime.datetime.now()
    today = today_now.strftime('%Y-%m-%d')

    return render_template('modify.html', prompt=prompt, content=content, image_url=image_url, today=today, summary=summary, tags=tags)

@var_blue.route('/save', methods=['POST'])
def save_modify_content():
    title = request.form['title']
    prompt = request.form['prompt']

    tags=[]
    tags_length = int(request.form['tag_length'])
    for i in range(tags_length):
        tags.append(request.form['tag'+str(i)])

    summary = request.form['summary']
    content = request.form['content']

    data = {'title' : title, 'tags' : tags, 'summary' : summary, 'content' : content}
    local_IO.modify_csv(data, prompt)

    tistory_post.post_to_blog(prompt, blog_key=key.access_token, blog_name=key.blog_name)

    return render_template('result.html', data=data)


import pygame

WallCors = []
class Character:

	StartingPositionX = 0
	StartingPositionY = 0
	Speed = 0
	size = [0,0]
	position = dict(x = StartingPositionX,y= StartingPositionY)

	def __init__(self):
		StartingPositionX = self.StartingPositionX
		StartingPositionY = self.StartingPositionY
		position = self.position

	@property
	def Position(self):
		return self.position

	@Position.setter
	def Position(self,x,y):
		self.position = [x,y]

	@property
	def Size(self):
		return self.size

	@property
	def Tile(self):
		return pygame.Rect(self.Position['x'],self.Position['y'],*self.Size)

	def DrawOn(self,surface):
		pygame.draw.rect(surface,(225, 225, 0), [self.Position['x'],self.Position['y'],*self.Size])

	def _checkObstacles(self,direction):
		if direction is 'left':
			if (self.Position['x'] - self.Speed) not in WallCors:
				return True

		if direction is 'right':
			if (self.Position['x'] +  self.Speed) not in WallCors:
				return True

		if direction is 'up':
			if (self.Position['y'] - self.Speed) not in WallCors:
				return True

		if direction is 'down':
			if (self.Position['y'] + self.Speed) not in WallCors:
				return True

	def GoLeft(self):
		if self._checkObstacles('left'):
			self.Position['x'] -=self.Speed
			self.Tile.move_ip(0-self.Speed,0)

	def GoRight(self):
		if self._checkObstacles('right'):
			self.Position['x'] +=self.Speed
			self.Tile.move_ip(0+self.Speed,0)

	def GoUp(self):
		if self._checkObstacles('up'):
			self.Position['y'] -=self.Speed
			self.Tile.move_ip(0,0-self.Speed)

	def GoDown(self):
		if self._checkObstacles('down') :
			self.Position['y'] +=self.Speed
			self.Tile.move_ip(0,0+self.Speed)


import logging
from typing import List

from tests.testing_utils.testing_classes import TestResult, NodeOutput

logger = logging.getLogger("Common Checks")

def common_checks(
        test_name: str,
        *args
):
    """
    Parses through the *args (should all be nodes) and "sorts" them such that the first return is a server,
        and all the rest are clients.

    Also makes sure that the cerr for all the nodes is empty
    :param test_name:
    :param args:
    :return:
    """
    test_name = test_name.split("/")[-1].split(".")[0]
    nodes = args
    server = None
    clients = []
    for maybe_node in nodes:
        if not isinstance(maybe_node, NodeOutput):
            continue
        if maybe_node.name == "Server":
            server = maybe_node
        else:
            clients.append(maybe_node)
        script_name = maybe_node.script_name

        if maybe_node.script_name != test_name:
            e_msg = f"{test_name} was loaded but script name was: {script_name}"
            logger.info(e_msg)
            return TestResult(script_name, None,
                              msg=e_msg,
                              error_flag=True)
        if maybe_node.cerr:
            e_msg = f"{test_name} {maybe_node.name} cerr was not empty. Cerr: {maybe_node.cerr}"
            logger.info(e_msg)
            return TestResult(script_name, False, msg=e_msg, err_context=maybe_node.cerr)
    return server, clients

def clear_empty_lines(to_clear) -> List:
    if isinstance(to_clear, str):
        to_clear = to_clear.split("\n")
    _holder = []
    for ln in to_clear:
        ln = ln.strip()
        if ln:
            _holder.append(ln)
    return _holder


#Head Material
#leftPorts=1
#rightPorts=1
#type=inputInterface
#done

#Block Description
'''
Author: Kyle Norland
Date: 10/13/20
Description: General signal reader from CSV
'''
#---------------------------------
#-------------Imports---------
#---------------------------------
import zmq
import json
import time
import sys

#Special
import csv

#Defaults:
proxyInputPort = 8000
proxyOutputPort = 9000
subTopics = []
pubTopics = ['cats']
blockName = "source--"

logfile = open("blockLogs/readCSVLog.txt", "w")
logfile.write("Got to here")
#------------------
#--Pull arguments--
#------------------
numArgs = len(sys.argv)
print("There are " + str(numArgs) + " arguments")

if numArgs >= 3:
    proxyInputPort = int(sys.argv[1])
    proxyOutputPort = int(sys.argv[2])
    stringArchitecture = sys.argv[3]
    try:
        print(stringArchitecture)
        jsonArch = json.loads(stringArchitecture)
        subTopics = jsonArch['subTopics']
        pubTopics = jsonArch['pubTopics']
        blockName = jsonArch['blockName']
        inputFileName = jsonArch['inputFiles'][0]
        print("The csv input file is: " + inputFileName)
    except:
        print("Something in the json loading process broke")
#------------------

#---------------------------------------
#-------Connect to the sockets----------
#---------------------------------------
context = zmq.Context()

pubSocket = context.socket(zmq.PUB)
pubSocket.connect("tcp://127.0.0.1:" + str(proxyInputPort))

subSocket = context.socket(zmq.SUB)
subSocket.connect("tcp://127.0.0.1:" + str(proxyOutputPort))

for subTopic in subTopics:
    subSocket.setsockopt(zmq.SUBSCRIBE, subTopic.encode())

#-------------------------------------------------------------
#-------------------Run Loop---------------------------
#------------------------------------------------------
logfile.write("Publishing to: " + str(pubTopics[0]))

with open(inputFileName, "r") as inputFile:
    csv_reader = csv.reader(inputFile, delimiter=",")
    line_count = 0

    dataLabels = []
    sendTemplate = {}
    for row in csv_reader:
        if line_count == 0:
            # Initialize the first row as the json keys
            for i in range(0, len(row)):
                dataLabels.append(str(row[i]))
                sendTemplate[str(row[i])] = ' '
            print(dataLabels)
            print(sendTemplate)
            line_count += 1

        if line_count != 0:
            #Create a json object and publish it.
            sendData = sendTemplate
            for i in range(0, len(row)):
                sendData[dataLabels[i]] = row[i]

            print(sendData)
            #Publish it
            pubSocket.send_string(pubTopics[0], zmq.SNDMORE)
            pubSocket.send_json(sendData)
            time.sleep(0.5)

            line_count += 1


import math
import heapq
import numpy as np
import random
from collections import namedtuple, deque

from model import QNetwork

import torch
import torch.nn.functional as F
import torch.optim as optim

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

class experience:
    """Represents an experience.

    It provides the __lt__ method so that it can be added to a heapq.
    """

    def __init__(self, state, action, reward, next_state, done, error=1.0):
        self.state = state
        self.action = action
        self.reward = reward
        self.next_state = next_state
        self.done = done
        self.error = error

    def __lt__(self, other):
        return self.error < other.error

    def __eq__(self, other):
        return self.error == other.error

    def __repr__(self):
        return str(self.error)

class ReplayBuffer:
    """Fixed-size buffer to store experience tuples."""

    def __init__(self, action_size, state_size, buffer_size, batch_size, seed):
        """Initialize a ReplayBuffer object.

        Params
        ======
            action_size (int): dimension of each action
            state_size (int): dimension of each state
            buffer_size (int): maximum size of buffer
            batch_size (int): size of each training batch
            seed (int): random seed
        """
        self.action_size = action_size
        self.state_size = state_size
        self.memory = deque(maxlen=buffer_size)
        self.batch_size = batch_size
        self.seed = random.seed(seed)

    def add(self, state, action, reward, next_state, done):
        """Add a new experience to memory."""
        e = experience(state, action, reward, next_state, done)
        self.memory.append(e)

    def sample(self):
        """Randomly sample a batch of experiences from memory."""
        experiences = random.sample(self.memory, k=self.batch_size)

        states = torch.from_numpy(np.vstack([e.state for e in experiences if e is not None])).float().to(device)
        actions = torch.from_numpy(np.vstack([e.action for e in experiences if e is not None])).long().to(device)
        rewards = torch.from_numpy(np.vstack([e.reward for e in experiences if e is not None])).float().to(device)
        next_states = torch.from_numpy(np.vstack([e.next_state for e in experiences if e is not None])).float().to(device)
        dones = torch.from_numpy(np.vstack([e.done for e in experiences if e is not None]).astype(np.uint8)).float().to(device)
        weights = torch.ones(self.batch_size, 1, dtype=torch.float, device=device)

        return (states, actions, rewards, next_states, dones, weights, None)

    def update(self, entries, errors):
        pass

    def __len__(self):
        """Return the current size of internal memory."""
        return len(self.memory)

class PrioritizedReplayBuffer:
    """Fixed-size buffer to store experience tuples."""

    def __init__(self, action_size, state_size, buffer_size, batch_size, seed, update_buffer_steps, alpha, beta, beta_inc):
        """Initialize a ReplayBuffer object.

        Params
        ======
            action_size (int): dimension of each action
            state_size (int): dimension of each state
            buffer_size (int): maximum size of buffer
            batch_size (int): size of each training batch
            seed (int): random seed
            update_buffer_steps (int): How often to update the buffer
            alpha (float): The priority exponent
            beta (float): The importance sampling exponent
            beta_inc (float): The importance sampling exponent increment
        """
        self.action_size = action_size
        self.state_size = state_size
        self.buffer_size = buffer_size
        self.batch_size = batch_size
        self.memory = []
        self.step_counter = 0
        self.seed = random.seed(seed)
        self.update_buffer_steps = update_buffer_steps
        self.alpha = alpha
        self.beta = beta
        self.beta_inc = beta_inc

    def add(self, state, action, reward, next_state, done, error=1.0):
        """Add a new experience to memory."""
        e = experience(state, action, reward, next_state, done, error)
        if len(self.memory) >= self.buffer_size:
            heapq.heapreplace(self.memory, e)
        else:
            heapq.heappush(self.memory, e)

        # sort memory
        self.step_counter = (self.step_counter + 1) % self.update_buffer_steps
        if self.step_counter == 0:
            heapq.heapify(self.memory)
            self.beta = min(1.0, self.beta * self.beta_inc)

    def sample(self):
        """Randomly sample a batch of experiences from memory."""
        states = np.zeros((self.batch_size, self.state_size), dtype=np.float)
        actions = np.zeros((self.batch_size,1), dtype=np.int32)
        rewards = np.zeros((self.batch_size,1), dtype=np.float)
        next_states = np.zeros((self.batch_size, self.state_size), dtype=np.float)
        dones = np.zeros((self.batch_size,1), dtype=np.uint8)
        p_j = np.zeros((self.batch_size,1), dtype=np.float)
        entries = []

        segment_size = len(self.memory) // self.batch_size
        for i in range(self.batch_size):
            start = i * segment_size
            end = len(self.memory) if i==self.batch_size-1 else (i+1)*segment_size
            j = random.randint(start, end-1)
            e = self.memory[j]
            entries.append(e)
            states[i,:] = e.state
            actions[i] = e.action
            rewards[i] = e.reward
            next_states[i,:] = e.next_state
            dones[i] = e.done
            p_j[i] = e.error

        # place tensors in GPU for faster calculations
        states = torch.from_numpy(states).float().to(device)
        actions = torch.from_numpy(actions).long().to(device)
        rewards = torch.from_numpy(rewards).float().to(device)
        next_states = torch.from_numpy(next_states).float().to(device)
        dones = torch.from_numpy(dones).float().to(device)
        p_j = torch.from_numpy(p_j).float().to(device)
        n = torch.tensor(self.batch_size, dtype=torch.float, device=device)
        alpha = torch.tensor(self.alpha, dtype=torch.float, device=device)
        beta = torch.tensor(self.beta, dtype=torch.float, device=device)

        # the math
        p_j = p_j ** alpha
        p_j = p_j / p_j.sum()
        weights = (n * p_j) ** -beta
        weights /= weights.max()

        return (states, actions, rewards, next_states, dones, weights, entries)

    def update(self, entries, errors):
        errors = errors.cpu()
        for entry, error in zip(entries, errors):
            entry.error = abs(error[0])

    def __len__(self):
        """Return the current size of internal memory."""
        return len(self.memory)


arr = [1, 1, 2, 3]
diff = 1

'''
    let the sum of 2 subset(scooped out of given subset) be S1 and S2.
    As diff is given and sum of S can calculated...
        S1 - S2 = diff
        S1 + S2 = sum
    By solving both, we can conclude
        S1 = (diff + sum)/2
    Hence, S1 as W(total weight of knapsack) in count of subset sum can
    give us the desired output.
'''

def countOfSubsetSum(arr, W, n):
    dp = [[0 for i in range(W+1)] for j in range(n+1)]

    for i in range(n+1):
        dp[i][0] = 1

    for i in range(1, n+1):
        for j in range(1, W+1):
            dp[i][j] = dp[i-1][j]

            if (arr[i-1] <= j):
                dp[i][j] += dp[i-1][j-arr[i-1]]
    return dp[n][W]

def countMin(arr, diff):
    su = (diff + sum(arr))//2
    return countOfSubsetSum(arr, su, len(arr))

print(countMin(arr, diff))


#Kullanıcıdan girilen sayının EBOB ve EKOK'unu bulan programı yazınız.

#Kullanıcıdan sayıları alalım:
sayi1 = int(input("Birinci sayıyı girin: "))
sayi2 = int(input("İkinci sayıyı girin: "))

#EBOB ve EKOK'u hesaplayalım:
a = sayi1
b = sayi2

while b:
    a, b = b, a % b

ebob_sonuc = a
ekok_sonuc = (sayi1 * sayi2) // ebob_sonuc
print(f"{sayi1} ve {sayi2} için EBOB: {ebob_sonuc}")
print(f"{sayi1} ve {sayi2} için EKOK: {ekok_sonuc}")


import sys
from time import sleep

import pygame

from bullet import Bullet
from alien import Alien

from alien_creator import *

def check_events(ai_settings, screen, stats, sb, play_button, ship, aliens, bullets):
    """ 响应 """
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            sys.exit()
        elif event.type == pygame.KEYDOWN:
            check_keydown_events(event, ai_settings, screen, stats, sb, play_button, ship, aliens, bullets)
        elif event.type ==pygame.KEYUP:
            check_keyup_events(event, ship)
        elif event.type == pygame.MOUSEBUTTONDOWN:
            mouse_x, mouse_y = pygame.mouse.get_pos()
            check_click_button(ai_settings, screen, stats, sb, play_button, ship, aliens, bullets, mouse_x, mouse_y)

def check_keydown_events(event, ai_settings, screen, stats, sb, play_button, ship, aliens, bullets):

    if event.key == pygame.K_UP:
        ship.moving_up = True
    elif event.key == pygame.K_RIGHT:
        ship.moving_right = True
    elif event.key == pygame.K_DOWN:
        ship.moving_down = True
    elif event.key == pygame.K_LEFT:
        ship.moving_left = True
    elif event.key == pygame.K_SPACE:
        if not stats.game_active:
            start_new_game(ai_settings, screen, stats, sb, play_button, ship, aliens, bullets)
        else:
            fire_bullet(ai_settings, screen, ship, bullets)
    elif event.key == pygame.K_RETURN or event.key == pygame.K_KP_ENTER:
        if not stats.game_active:
            start_new_game(ai_settings, screen, stats, sb, play_button, ship, aliens, bullets)
        #else:
        #    fire_bullet(ai_settings, screen, ship, bullets)
    elif event.key == pygame.K_q: # event.key == pygame.K_ESCAPE:
        sys.exit()

def check_keyup_events(event, ship):

    if event.key == pygame.K_UP:
        ship.moving_up = False
    elif event.key == pygame.K_RIGHT:
        ship.moving_right = False
    elif event.key == pygame.K_DOWN:
        ship.moving_down = False
    elif event.key == pygame.K_LEFT:
        ship.moving_left = False

def check_click_button(ai_settings, screen, stats, sb, play_button, ship, aliens, bullets, mouse_x, mouse_y):
    play_button_clicked = play_button.rect.collidepoint(mouse_x, mouse_y)
    if play_button_clicked and not stats.game_active:
        start_new_game(ai_settings, screen, stats, sb, play_button, ship, aliens, bullets)

def start_new_game(ai_settings, screen, stats, sb, play_button, ship, aliens, bullets):
    ai_settings.initialize_dynamic_settings()

    stats.reset_stats()
    stats.game_active = True

    sb.prep_all()

    aliens.empty()
    bullets.empty()

    create_fleet(ai_settings, stats, screen, ship, aliens)
    ship.center_ship()

def update_screen(ai_settings, screen, stats, sb, ship, aliens, bullets, play_button):
    # repaint
    screen.fill(ai_settings.bg_color)
    for bullet in bullets.sprites():
        bullet.draw_bullet()
    ship.blitme()
    for alien in aliens.sprites():
        alien.blitme()
    # aliens.draw(screen)

    sb.show_score()

    if not stats.game_active:
        play_button.draw_button()

    # visual
    pygame.display.flip()

def update_bullets(ai_settings, screen, stats, sb, ship, aliens, bullets):
    """ """
    bullets.update()
    for bullet in bullets.copy():
        if bullet.rect.bottom <= 0:
            bullets.remove(bullet)

    check_bullet_alien_collisions(ai_settings, screen, stats, sb, ship, aliens, bullets)

def fire_bullet(ai_settings, screen, ship, bullets):
    """ create a bullet, add it into group"""
    if len(bullets) < ai_settings.bullet_allowed:
            new_bullet = Bullet(ai_settings, screen, ship)
            bullets.add(new_bullet)

def check_bullet_alien_collisions(ai_settings, screen, stats, sb, ship, aliens, bullets):
    collisions = pygame.sprite.groupcollide(bullets, aliens, True, True)
    if collisions:
        for aliens in collisions.values():
            stats.score += ai_settings.alien_points * len(aliens)
        sb.prep_score()
    if len(aliens) == 0:
        refresh_next_fleet(ai_settings, screen, stats, sb, ship, aliens, bullets)

def check_fleet_edges(ai_settings, aliens):
    for alien in aliens.sprites():
        if alien.check_edges():
            change_fleet_direction(ai_settings, aliens)
            break

def change_fleet_direction(ai_settings, aliens):
    for alien in aliens.sprites():
        alien.rect.y += ai_settings.fleet_drop_speed
    ai_settings.fleet_direction *= -1

def check_aliens_bottom(ai_settings, stats, screen, ship, aliens, bullets):
    screen_rect = screen.get_rect()
    for alien in aliens.sprites():
        if alien.rect.bottom >= screen_rect.bottom:
            ship_hit(ai_settings, stats, screen, ship, aliens, bullets)
            break

def ship_hit(ai_settings, stats, screen, ship, aliens, bullets):

    if stats.ships_left > 0:
        stats.ships_left -= 1

        aliens.empty()
        bullets.empty()

        sleep(1.0)

        create_fleet(ai_settings, stats, screen, ship, aliens)
        ship.center_ship()

    else:
        stats.game_active = False

def update_aliens(ai_settings, stats, screen, ship, aliens, bullets):
    check_fleet_edges(ai_settings, aliens)
    aliens.update()
    if pygame.sprite.spritecollideany(ship, aliens):
        ship_hit(ai_settings, stats, screen, ship, aliens, bullets)
    check_aliens_bottom(ai_settings, stats, screen, ship, aliens, bullets)

def refresh_next_fleet(ai_settings, screen, stats, sb, ship, aliens, bullets):
    bullets.empty()
    stats.level += 1
    if stats.level>3 and stats.level%5!=0:
        ai_settings.increase_speed()
    sb.prep_level()
    create_fleet(ai_settings, stats, screen, ship, aliens)


from pe_utils import *

i = 232000000
running = True

while running:
	j = 0
	for j in range(1, 21, 1):
		if not divisible(i, j):
			i += 1
			print(i, " : failed")
			break
		else:
			if j == 20:
				print(i, " : worked")
				running = False
				break


import torch.nn as nn
from mmcv.cnn import build_norm_layer

from ..builder import NECKS

@NECKS.register_module()
class Feature2Pyramid(nn.Module):
    """Feature2Pyramid.

    A neck structure connect ViT backbone and decoder_heads.

    Args:
        embed_dims (int): Embedding dimension.
        rescales (list[float]): Different sampling multiples were
            used to obtain pyramid features. Default: [4, 2, 1, 0.5].
        norm_cfg (dict): Config dict for normalization layer.
            Default: dict(type='SyncBN', requires_grad=True).
    """

    def __init__(self,
                 embed_dim,
                 rescales=[4, 2, 1, 0.5],
                 norm_cfg=dict(type='SyncBN', requires_grad=True)):
        super(Feature2Pyramid, self).__init__()
        self.rescales = rescales
        self.upsample_4x = None
        for k in self.rescales:
            if k == 4:
                self.upsample_4x = nn.Sequential(
                    nn.ConvTranspose2d(
                        embed_dim, embed_dim, kernel_size=2, stride=2),
                    build_norm_layer(norm_cfg, embed_dim)[1],
                    nn.GELU(),
                    nn.ConvTranspose2d(
                        embed_dim, embed_dim, kernel_size=2, stride=2),
                )
            elif k == 2:
                self.upsample_2x = nn.Sequential(
                    nn.ConvTranspose2d(
                        embed_dim, embed_dim, kernel_size=2, stride=2))
            elif k == 1:
                self.identity = nn.Identity()
            elif k == 0.5:
                self.downsample_2x = nn.MaxPool2d(kernel_size=2, stride=2)
            elif k == 0.25:
                self.downsample_4x = nn.MaxPool2d(kernel_size=4, stride=4)
            else:
                raise KeyError(f'invalid {k} for feature2pyramid')

    def forward(self, inputs):
        assert len(inputs) == len(self.rescales)
        outputs = []
        if self.upsample_4x is not None:
            ops = [
                self.upsample_4x, self.upsample_2x, self.identity,
                self.downsample_2x
            ]
        else:
            ops = [
                self.upsample_2x, self.identity, self.downsample_2x,
                self.downsample_4x
            ]
        for i in range(len(inputs)):
            outputs.append(ops[i](inputs[i]))
        return tuple(outputs)


from django.shortcuts import render
from .models import Pilotos, Corridas, Circuitos

# Create your views here.

def index(request):
    return render(request, 'index.html')

def detalhe_pilotos(request, pk):
    pilotos = Pilotos.objects.get(pk=pk)
    pilotos_quantidade = Pilotos.objects.all()
    context = {
        'sobrenome': pilotos.ultimo_nome,
        'nome': pilotos.primeiro_nome,
        'referencia': pilotos.ref_piloto,
        'nascimento': pilotos.nascimento,
        'nacionalidade': pilotos.nacionalidade_piloto,
        'total': pilotos_quantidade.count,
    }
    return render(request, 'pilotos.html', context)

def detalhe_corridas(request, pk):
    corridas = Corridas.objects.get(pk=pk)
    corrida_fk = corridas.id_circuito;
    corridas_quantidade = Corridas.objects.all()
    context = {
        'ano': corridas.ano_corrida,
        'nome_circuito': corrida_fk.nome_circuito,
        'rodada': corridas.rodada,
        'nome': corridas.nome_corrida,
        'data': corridas.data_corrida,
        'tempo': corridas.tempo_corrida,
        'total': corridas_quantidade.count
    }
    return render(request, 'corridas.html', context)

def detalhe_piloto_por_nome(request, nome):
    pilotos = Pilotos.objects.get(primeiro_nome=nome)
    pilotos_quantidade = Pilotos.objects.all()
    context = {
        'sobrenome': pilotos.ultimo_nome,
        'nome': pilotos.primeiro_nome,
        'referencia': pilotos.ref_piloto,
        'nascimento': pilotos.nascimento,
        'nacionalidade': pilotos.nacionalidade_piloto,
        'total': pilotos_quantidade.count

    }
    return render(request, 'pilotos.html', context)

def pilotos(request):
    pilotos= Pilotos.objects.all()
    context = {
        'pilotos': pilotos
    }
    return render(request, 'pilotos.html', context)


# Python 3.6 introduced a new implementation of dict.
# dict now keeps its items ordered as well.

# An OrderedDict is a dictionary subclass that remembers the order in which its contents are added.
from collections import OrderedDict

if __name__ == "__main__":
    order_dic = OrderedDict()
    order_dic['a'] = 'A'
    order_dic['b'] = 'B'
    order_dic['c'] = 'C'
    order_dic['d'] = 'D'
    order_dic['e'] = 'E'

    for k, v in order_dic.items():
        print(k, v)

    # https://docs.python.org/3/library/collections.html#collections.OrderedDict.popitem
    # last=True -> LIFO
    # last=False -> FIFO

    print(
        order_dic.popitem(last=True) # -> ('e', 'E')
    )

    print(
        order_dic.popitem(last=False) # -> ('a', 'A')
    )

    # https://docs.python.org/3/library/collections.html#collections.OrderedDict.move_to_end
    # last=True   -> The item is moved to the right end
    # last=False  -> The item is moved to the beginning

    print('origin:', order_dic) # OrderedDict([('b', 'B'), ('c', 'C'), ('d', 'D')])
    order_dic.move_to_end('b', last=True)
    print(order_dic) # OrderedDict([('c', 'C'), ('d', 'D'), ('b', 'B')])
    order_dic.move_to_end('b', last=False)
    print(order_dic) # OrderedDict([('b', 'B'), ('c', 'C'), ('d', 'D')])


from tkinter import*
from typing import List

# https://m.blog.naver.com/amethyst_lee/222012497834
root = Tk()
root.title("GUI test")
root.geometry("640x700")

def btncmd():
    print("버튼 클릭됨")
    print(txt.get("1.0", END)) # 첫번째 라인, 0:0번째 컬럼위치
    print(ent.get())
    txt.delete("1.0", END)
    ent.delete(0, END)

def change():
    label.config(text = "변경됨")

def orderGet():
    print(burger_var.get())

def printList():
    print(listbox.curselection())

btn1 = Button(root, text = "버튼1", command=change)
btn1.pack()

btn2 = Button(root, padx=10, pady=10, text = "버튼2", command = btncmd)
btn2.pack()

label = Label(root, text = "변경 전 텍스트")
label.pack()

txt = Text(root, width=30, height=10)
txt.pack()
txt.insert(END, "글자를 입력하세요")

ent = Entry(root, width=30)
ent.pack()

burger_var = IntVar()
btn_b1 = Radiobutton(root, text = "b1", value=1, variable=burger_var)
btn_b1.select()
btn_b2 = Radiobutton(root, text = "b2", value=2, variable=burger_var)
btn_b3 = Radiobutton(root, text = "b3", value=3, variable=burger_var)
btn_b4 = Radiobutton(root, text = "b4", value=4, variable=burger_var)

btn_b1.pack()
btn_b2.pack()
btn_b3.pack()
btn_b4.pack()

orderBtn = Button(root, text="order", command=orderGet)
orderBtn.pack()

cb1 = Checkbutton(root, text = "cb1")
cb1.pack()

cb2 = Checkbutton(root, text = "cb2")
cb2.pack()

listbox = Listbox(root, selectmode="extended", height = 0)
listbox.insert(0, "학교홈피")
listbox.insert(1, "명대신문")
listbox.insert(2, "도서관")
listbox.insert(3, "44")
listbox.insert(4, "55")
listbox.pack()

listBtn = Button(root, text="listBtn", command=printList)
listBtn.pack()

root.mainloop()


es_igual_que = 5 == 6

es_distinto = 5 != 6

mayor_que = 5 > 6

mayor_igual = 5 >= 6

menor_que = 5 < 6

menor_igual = 5 <= 6

#calculos combinados
a = 5
b = 10
c = 20

comparacion = a + b < c

#comparacion de datos
contraseña_almacenada = "topito"
contraseña_ingresada = "aloja"

print(contraseña_almacenada == contraseña_ingresada)


class Solution(object):
    def smallerNumbersThanCurrent(self, nums):
        """
        :type nums: List[int]
        :rtype: List[int]
        """
        answer = []
        for i in range(len(nums)):
            count = 0
            for j in range(len(nums)):
                if nums[i] > nums[j] and j != i:
                    count += 1
            answer.append(count)
        return answer

Easy/1365. How Many Numbers Are Smaller Than the Current Number.py


#Version1 Lab03
#Version2 added the tuple however did not use in lab
"""coins = [
('half-dollar', 50)
('quarter', 25),
('dime', 10),
('nickel', 5),
('penny', 1)
]"""
#coin names and their amounts
quarter = 25
dime = 10
nickel = 5
penny = 1

#defining the variable to be converted
x = input("Enter a dollar amount: ")
x = float(x)
#formula to calculate variable to convertable amount
y = x*100

#separate equations to using floor division // and modulus % to convert dollar *100 to coins
#if y >= 25:
quarters = y//quarter
d = y%quarter
dimes = d//dime
n = d%dime
nickels = n//nickel
p = n%nickel
pennies = p//penny
message = (f'There are {quarters} quarters, {dimes} dimes, {nickels}, nickels, and {pennies} pennies')

print(message)


txt = open("d3.txt").read()
data = txt.split("\n")
triangles = []
for item in data:
	triangles.append(item.split("  ")[1:])
triangles.pop()
count = 0
for triangle in triangles:
	if '' in triangle:
		triangle.remove('')
	for id, line in enumerate(triangle):
		triangle[id] = int(triangle[id].strip())

	if triangle[0] + triangle[1] > triangle[2] and\
	   triangle[0] + triangle[2] > triangle[1] and\
	   triangle[1] + triangle[2] > triangle[0]:
	   count += 1

print(count)


# This problem deals with family trees
# but I didn't raelise that at first
# so I asked
# what if the trees have cycles in it

with open("inheritance.txt") as f:
    N = int(f.readline())
    tree = {}
    dead = set()

    # build adjacency list
    for i in range(N):
        children = f.readline().split(":")
        node = children[0]

        # format node to get rid of extra info
        if node[-1] == "x":
            node = node[:-1]
            dead.add(node)
        elif node[-1] == "*":
            node = node[:-1]

        tree[node] = children[1].split()

def rearrange(l):
  return [x for x in l if x[0] == "M"] + \
         [x for x in ["+".join([x for x in l if x[0] == "F"])] if x != ""]

for node in tree:
  tree[node] = rearrange(tree[node])

# generator functions
def preOrder(n):
    yield n
    for c in tree[n]:
        if c[0] == "F":
            yield c
        else:
            yield from preOrder(c)

def postOrder(n):
  for c in tree[n]:
    if c[0] == "F":
      yield c
    else:
      yield from postOrder(c)

# output results
for n in preOrder("M0"):
    if n[0] == "F" or n in dead:
      continue
    else:
      print(n, end=" ")

for n in postOrder("M0"):
    if n[0] == "M" or n in dead:
      continue
    else:
      print(n, end=" ")

print()


# coding=utf-8

import xlrd

def read_xlrd(excelFile):
    data = xlrd.open_workbook(excelFile)
    table = data.sheet_by_index(0)
    dataFile = []

    for rowNum in range(table.nrows):
        # if 去掉表头
        if rowNum > 0:
            dataFile.append(table.row_values(rowNum))

    return dataFile

if __name__ == '__main__':
    excelFile = 'C:\\Users\Administrator\Desktop\Python代码练习\excel与python交互/demo.xlsx'
    print(read_xlrd(excelFile=excelFile))


import ASUS.GPIO as GPIO
import time
import importlib

defaults = importlib.import_module('ircgpio.ircauxiliary.defaults')
parser = importlib.import_module('ircgpio.ircauxiliary.parser')

amendment = []
GPIO.setwarnings(False)
GPIO.setmode(GPIO.BOARD)
pin=7
ir_sensor_th = 36
GPIO.setup(pin, GPIO.IN)

def clearlst(lst, str):
	for l in lst:
		if l == "":
			lst.remove(str)

def getrcsubparam(configfile, rcname, subparam):
	rcontrols = parser.getvalfromconfigbykeyword(defaults.rc, defaults.keywords, configfile, [""])
	clearlst(rcontrols, "")

	data = parser.getvalfromconfigbykeyword(defaults.data, defaults.keywords, configfile, [""])
	clearlst(data, "")

	rc = parser.getvalfromlistbykeyword(rcname, rcontrols, data, [""])

	pairs = parser.getparametersandvals(rc, defaults.subparams, "=", [" ", "	"], ",")

	return parser.extractvaluebyparam(subparam, pairs)

def makecodespyfile(filename, dict, _len, buttons):
	f = open(filename,"w")
	f.write("code_len = ")
	f.write(str(_len))
	f.write("\n")
	f.write("btn_codes = {\n")
	i = 0
	for b in buttons:
		f.write("\'")
		f.write(b)
		f.write("\' ")
		f.write(": [")
		j = 0
		for c in dict.get(b):
			f.write(str(c))
			if(j != len(dict.get(b)) - 1):
				f.write(", ")
			else:
				f.write("]")
			j = j + 1
		if(i != len(buttons) - 1):
			f.write(",\n")
		else:
			f.write("\n}")
		i = i + 1

	f.close()

def count(lst):
	map = []
	for l in lst:
		fq = 0
		for r in lst:
			if l == r:
				fq = fq + 1
		map.append([l,fq])

	res = []

	for i in range (0,len(map)):
		if i == 0:
			res.append(map[i])
		else:
			if map[i] not in res:
				res.append(map[i])

	return res

def findcalibrparam(lst, th):
	max = [[0,0]]
	for l in lst:
		if l[1] > max[0][1]:
			max[0] = l

	if max[0][1] < th:
		return -1

	if max[0][1] >= th:
		return max[0][0]

def getmax(lst):
	max = 0
	for l in lst:
		if l > max:
			max = l

	return max

def dispersion(lst):
	max = getmax(lst)
	_disp = []

	for l in lst:
		if l >= 0.8 * max:
			_disp.append(max - l)

	disp = 0
	sum = 0
	for d in _disp:
		sum = sum + d

	disp = sum / len(_disp)
	_disp.clear()

	return disp

def calibration():
	buffer = []
	calibr = []
	print("Startting calibration")
	ic = 0
	while True:
		tic = time.time()
		GPIO.wait_for_edge(pin, GPIO.FALLING)
		toc = time.time()
		if (toc - tic < 1):
			if ((toc - tic)*1000 < ir_sensor_th):
				buffer.append(float("{0:.3f}".format((toc - tic)*1000)))
			else:
				print("Calibration step ", ic)
				calibr.append(len(buffer.copy()))
				buffer.clear()
				ic = ic + 1
				if ic == 11:
					ic = 0
					cparams = count(calibr.copy())
					print(cparams)
					calibr.clear()
					print("Start calibration again? (Y/any key)")
					inp = input()
					if inp == "Y":
						print("New calibration")
					else:
						cp = findcalibrparam(cparams, 5)
						if (cp == -1):
							print("Calibration parameter has hot been defined, starting calibration again")
						else:
							return cp

def dcalibration(p_calib):
	buffer = []
	avg = []
	ic2 = 0
	print("Startting double calibration")
	while True:
		tic = time.time()
		GPIO.wait_for_edge(pin, GPIO.FALLING)
		toc = time.time()
		if (toc - tic < 1):
			if ((toc - tic)*1000 < ir_sensor_th):
				buffer.append(float("{0:.3f}".format((toc - tic)*1000)))
			else:
				if (len(buffer) == p_calib):
					print("Double calibration step ", ic2)
					print (buffer)
					avg.append(buffer.copy()[0])
					ic2 = ic2 + 1
					if ic2 == 11:
						max = getmax(avg)
						disp = float("{0:.3f}".format(dispersion(avg)))
						avg.clear()
						res = [max, disp]
						return res

				buffer.clear()

def frecord(p_calib, irsteps, corr = [0,0]):
	buffer = []
	record = []
	if len(record) != 0:
		record.clear()
	print("Recording...")
	ir = 1
	while True:
		tic = time.time()
		GPIO.wait_for_edge(pin, GPIO.FALLING)
		toc = time.time()
		if (toc - tic < 1):
			if ((toc - tic)*1000 < ir_sensor_th):
				buffer.append(float("{0:.3f}".format((toc - tic)*1000)))
			else:
				max = getmax(buffer)
				if (len(buffer) == p_calib):
					if corr != [0,0]:
						print(corr)
						if max < corr[0] - corr[1]:
							continue
					print("Record step ", ir)
					print (buffer)
					record.append(buffer.copy())
					ir = ir + 1
					if ir == irsteps + 1:
						ir = 1
						tmp = []
						for i in range(0,p_calib):
							sum = 0
							for j in range (0, irsteps):
								sum = record[j][i] + sum
							tmp.append(float("{0:.3f}".format((sum)/irsteps)))
						print("Average")
						print(tmp)
						print("Save record? (Y/any key)")
						inp = input()
						if inp != "Y":
							print("New record")
							tmp.clear()
							record.clear()
						else:
							return tmp.copy()
				buffer.clear()

def savebutton(p_calib, buttons, corr = [0,0]):
	dict = {}
	for b in buttons:
		print("Record started for button ", b)
		code = frecord(p_calib, 3, corr)
		dict[b] = code
		print("Saved!")

	return dict

p_calib = calibration()
print("Calibration parameter = ", p_calib)

pairs = parser.getparametersandvals(parser.getvalfromconfigbykeyword(defaults.record,defaults.keywords,defaults.configuration,[""]), defaults.globparameters, "=", [" ", "	"], ",")
rec = parser.extractvaluebyparam(defaults.rcrec, pairs)

pairs = parser.getparametersandvals(parser.getvalfromconfigbykeyword(defaults.oglobal,defaults.keywords,defaults.configuration,[""]), defaults.globparameters, "=", [" ", "	"], ",")
_dc = parser.extractvaluebyparam(defaults.double_calibration, pairs)

irtorec = rec[0]
dc = _dc[0]

if dc == defaults.on:
	amendment = dcalibration(p_calib)
	print("Amendmebt parameter = ", amendment)

_buttonfile = getrcsubparam(defaults.configuration, irtorec, defaults.button)
_codesfile = getrcsubparam(defaults.configuration, irtorec, defaults.codes)
_folder = getrcsubparam(defaults.configuration, irtorec, defaults.folder)

buttonfile = _folder[0] + "/" + _buttonfile[0]
codesfile = _folder[0] + "/" + _codesfile[0]

buttons = parser.getvalfromconfigbykeyword(defaults.bu, defaults.subkeywords, buttonfile, [""])

if dc == defaults.on:
	dict = savebutton(p_calib, buttons, amendment)
else:
	dict = savebutton(p_calib, buttons)

makecodespyfile(codesfile, dict, p_calib, buttons)


from keras.models import Model
from keras.models import load_model
from keras.layers import Input, Convolution2D, MaxPooling2D, Dense, Dropout, Activation, Flatten
from keras.utils import np_utils # utilities for one-hot encoding of ground truth values
from CNNParameters import batch_size, num_epochs, kernel_size, pool_size, conv_depth_1, conv_depth_2, drop_prob_1, drop_prob_2, hidden_size
from sklearn import preprocessing
from keras.callbacks import ModelCheckpoint,EarlyStopping
import csv
import numpy as np
from resnet import ResnetBuilder

'''
	0. Make sure training and testing data are in ../data
	1. Run:
			load_x() to load training x
			load_y() to load training y
'''

#x_train_file = open('../data/sample1.csv')
#x_train_file = open('../data/train_x.csv')
#y_train_file = open('../data/train_y.csv')
#x_test_file = open('../data/test_x.csv')

#model output path : This stores the latest model
output_path = "resnet.h5"

#prediction output path : Final prediction for Kaggle submission
prediction_filepath = "resnet.prediction.csv"

#where we will save the weights
weights_filepath="resnet.weights.best.hdf5"

#This is the pipeline function from preprocessing
#returns (50000, 64, 64, 1) shape np array
def load_x():
	path = "../data/training_data.npy"
	training_data_x = np.load(path)
	training_data_x = training_data_x.reshape(50000, 64, 64, 1)

	print("Training data x: {}".format(training_data_x.shape))

	return training_data_x

def load_y():
	path = "../data/train_y.csv"
	training_data_y = np.loadtxt(path, delimiter="\n")
	#Label Encode our results.
	#[  0.   1.   2.   3.   4.   5.   6.   7.   8.   9.  10.  11.  12.  13.  14.
	#15.  16.  17.  18.  20.  21.  24.  25.  27.  28.  30.  32.  35.  36.  40.
	#42.  45.  48.  49.  54.  56.  63.  64.  72.  81.]
	#will be encoded to 0,1,2,3,...
	#this means that "81" will be [0,0,0,0,0,...,0,1] where the 40th entry is 1
	le = preprocessing.LabelEncoder()

	le.fit(training_data_y)		#fit with label encoder
	training_data_y = le.transform(training_data_y)		#encode with label encoder

	#one-hot encoding
	training_data_y = np_utils.to_categorical(training_data_y)

	print("Training data y: {}".format(training_data_y.shape))

	return training_data_y, le

def load_test():
	path = "../data/test_data.npy"
	test_data_x = np.load(path)
	test_data_x = test_data_x.reshape(10000, 64, 64, 1)

	print("Test data x: {}".format(test_data_x.shape))

	return test_data_x

def build_resnet():
	input_shape = (1, 64, 64)
	output_shape = 40

	model = ResnetBuilder.build_resnet_18(input_shape,output_shape)
	model.compile(loss='categorical_crossentropy', # using the cross-entropy loss function
				  optimizer='adam', # using the Adam optimiser
				  metrics=['accuracy']) # reporting the accuracy
	return model

def train_resnet():
	model = build_resnet()
	x_train = load_x()

	y_train,a = load_y()

	checkpoint = ModelCheckpoint(weights_filepath, monitor='acc', verbose=1, save_best_only=True, mode='max')
	stopping = EarlyStopping(monitor='acc', min_delta=0.001, patience=10, verbose=1, mode='auto')
	callbacks_list = [checkpoint]

	print("Training ResNet")
	model.fit(x_train, y_train,                # Train the model using the training set...
			  batch_size=32, epochs=10,
			  verbose=1,callbacks = callbacks_list, validation_split=0.1) # ...holding out 10% of the data for validation

#Predicts our test set and outputs results in csv format
def predict(model, label_encoder):

	labels = label_encoder.classes_

	#load our test set
	x_test = load_test()

	#use model to predict our test set
	#this contains probabilities of each class
	pred = model.predict(x_test,verbose = 1)

	#this transforms pred to hold indices of the classes
	result = []
	for i in range(pred.shape[0]):
		label = labels[np.argmax(pred[i])]
		result.append(label)

	result2 = list(map(lambda x : int(x), result))

	#save to csv
	with open(prediction_filepath, 'w') as f:
		writer = csv.writer(f)
		writer.writerow(('Id', 'Label'))
		for i in range(1,len(result2)+1):
			writer.writerow((i, result2[i-1]))

	print("\nPredictions Successfully saved in ./{}".format(prediction_filepath))

if __name__ == '__main__':
	#train_resnet()
	a,le = load_y()
	print("loading model")
	model = build_resnet()
	resnet = model.load_weights("resnet.weights.best.hdf5")

	predict(resnet, le)


from time import sleep
import random
import yaml
from threading import Thread
import logging
import threading

import common
import cecclient
import mqttclient

def init(config):
    logging.basicConfig(filename='stereo.log', level=logging.INFO, format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')
    config_file = open('config.yaml', 'r')
    config = yaml.load(config_file, Loader=yaml.FullLoader)
    config_file.close()
    return config

def publish_state():
    mymqttclient.client.publish(common.power_avail_topic, payload='online')
    mycecclient.request_stereo_power()
    mycecclient.request_audio_status()
    sleep(0.5)
    power = mycecclient.get_stereo_power()
    volume = mycecclient.get_volume()
    logging.debug("power status: %s", power)
    logging.debug("volume status: %s", volume)
    if power == 0:
        mymqttclient.client.publish(common.power_state_topic, payload='OFF')
    elif power == 1:
        mymqttclient.client.publish(common.power_state_topic, payload='ON')
    else:
        logging.error("Uh oh, weird state!")
    mymqttclient.client.publish(common.volume_state_topic, payload=str(volume))
    sleep(15)

def stereo_status_worker():
    while True:
         publish_state()

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        logging.info("MQTT connected to host "+mymqttclient.broker+":1883"+" rc: "+str(rc))
        mymqttclient.client.publish(common.power_avail_topic, payload='online')
        mymqttclient.client.subscribe(common.power_command_topic)
        mymqttclient.client.subscribe(common.source_command_topic)
        mymqttclient.client.subscribe(common.volume_command_topic)
        publish_state()
    else:
        logging.error("MQTT failed to connect to host "+mymqttclient.broker+":1883"+" with rc: "+str(rc))

def on_disconnect(client, userdata, rc):
    logging.warning("Power host connection lost with rc: "+str(rc))

def on_message(client, userdata, msg):
    if msg.topic == common.power_command_topic:
        if msg.payload.decode() == 'ON':
            mymqttclient.client.publish(common.power_state_topic, payload='ON')
            mycecclient.stereo_power_on()
        elif msg.payload.decode() == 'OFF':
            mymqttclient.client.publish(common.power_state_topic, payload='OFF')
            mycecclient.stereo_power_off()
    elif msg.topic == common.source_command_topic:
        if msg.payload.decode() == 'Airplay':
            mymqttclient.client.publish(common.source_state_topic, payload='Airplay')
            mycecclient.source_airplay()
        elif msg.payload.decode() == 'TV':
            mymqttclient.client.publish(common.source_state_topic, payload='TV')
            mycecclient.source_tv()
    elif msg.topic == common.volume_command_topic:
        if msg.payload.decode() == 'ON':
            mymqttclient.client.publish(common.volume_state_topic, payload=(mycecclient.get_volume())+2)
            mycecclient.stereo_volume_up()
        elif msg.payload.decode() == 'OFF':
            mymqttclient.client.publish(common.volume_state_topic, payload=(mycecclient.get_volume())-2)
            mycecclient.stereo_volume_down()
    else:
        logging.error("bad command")

# main setup
config = None
mycecclient = None
mymqttclient = None

if __name__ == '__main__':
    config = init(config)
    mycecclient = cecclient.pyCecClient()
    mymqttclient = mqttclient.mqttClient(config)
    # setup mycecclient
    mycecclient.InitLibCec()
    # setup mymqttclient
    mymqttclient.set_connect_callback(on_connect)
    mymqttclient.set_disconnect_callback(on_disconnect)
    mymqttclient.set_message_callback(on_message)
    # stereo status monitoring thread setup
    stereoStatusThread = threading.Thread(target=stereo_status_worker)
    # start mqtt client
    mymqttclient.start_mqtt()
    # start stereo status monitoring thread
    sleep(2)
    stereoStatusThread.start()

    while True:
        sleep(0.1)


from abc import ABCMeta, abstractmethod;

print("This program is to test a inheritence in classes.");

# A Parent vehicle class
class Vehicle:

	__metaclass__ = ABCMeta;

	wheels = 0;

	def __init__(self, model, brand, color):
		self.model = model;
		self.brand = brand;
		self.color = color;

	def displaycar(self):
		#print(self.model, "is a ", self.color, "color", self.wheels, "wheeler light motor vehicle manufactured by ", self.make,".");
		print("%s is a %s color %d wheeler light motor vehicle manufactured by %s." % (self.model, self.color, self.wheels, self.brand));

class Car(Vehicle):

	wheels = 4;

class Truck(Vehicle):

	wheels = 6;

	def displaycar(self):
		print("This is truck's method.");
		super().displaycar();

bmw = Car("BMW", "BMW", "Red");
bmw.displaycar();

bharat = Truck("Bharat", "TaTa", "Black");
bharat.displaycar();

#c200 = Car("Mercedes", "Diamler", "Black");
#c200.displaycar();


__author__='Kalpak Seal'
'''
A small hash function
'''

def hash(astring, tablesize):
        sum = 0
        for pos in range(len(astring)):
                sum = sum + ord(astring[pos])

        return sum%tablesize

def main():
        arr = [x for x in input('Enter names to compute Hash ').split()]
        tablesize = int(input('Enter the table size '))

        for i in range(len(arr)):
                #print (arr[i])
                print (hash(arr[i], tablesize))

if __name__ == '__main__':
        main()


def calc(n):
    ret = 0
    for i, num in enumerate(reversed(x)):
        ret += (n**i) * int(num)
    return ret

def sovled():
    # Xの桁数がM以下の時は１種類しか存在しない
    if len(x) == 1:
        if int(x) <= m:
            print(1)
        else:
            print(0)
        return

    base = 0
    for i in range(10):
        if str(i) in x:
            base = i

    left, right = base, 1 << 100
    while right - left > 1:
        mid = (left + right) // 2
        # print(f'{left=}, {mid=}, {right=}')
        if calc(mid) <= m:
            left = mid
        else:
            right = mid

    print(left - base)

if __name__ == '__main__':
    x = input()
    m = int(input())

    sovled()

# TLE source code.
# def sovled():
#     if len(x) == 1:
#         print(m//int(x))
#         return

#     base = 0
#     for i in range(10):
#         if str(i) in x:
#             base = i + 1

#     cnt = 0
#     while True:
#         calc = 0
#         for i, num in enumerate(reversed(x)):
#             calc += (base**i) * int(num)
#         if calc > m:
#             break
#         cnt += 1
#         base += 1
#     print(cnt)

# if __name__ == '__main__':
#     x = input()
#     m = int(input())

#     sovled()


from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM
import numpy as np
from pytorch_transformers import BertConfig
import pandas as pd
import os
import argparse
import pickle
import platform

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--max_seq_length', type=int, default = 50,
                        help='length')
    # parser.add_argument('--split_path', type=str, default = "/processed/split_data.json",
    #                     help='length')
    parser.add_argument('--save_path', type=str, default = "twitter_data/bert_features",
                        help='length')
    return parser.parse_args()

def bert_feature(args, data, class_to_idx=None):
    path = args.path
    # data_path = os.path.join(path, "annotation/" + language + "/labeled/total.json")
    # save_path = os.path.join(path, "annotation/" + language + args.split_path)
    if platform.system() == 'Linux' or 'Darwin':
        tokenizer = BertTokenizer.from_pretrained(path + '/bert-base-uncased', do_lower_case=True)
    else:
        tokenizer = BertTokenizer.from_pretrained(path + '\\bert-base-uncased', do_lower_case=True)

    # with open(data_path,'r',encoding='utf8') as fp:
    #     data = json.load(fp)

    max_seq_length = args.max_seq_length
    features = {}
    Input_ids, Input_mask, Segment_ids = [], [], []
    Input_ids_dom, Input_mask_dom, Segment_ids_dom = [], [], []
    ids, users, Label = [], [], []
    harm, target, activation, polarity = [], [], [], []
    ment, tag = [], []
    raw_data = []
    max_len = 0
    dom = []
    dom_lab = []

    for index, tweet in data.iterrows():
        # tweet_contant, hash_tag, mention = remove_tag(tweet['data'], remain=False)
        # tweet_contant, _ = do_clean(tweet_contant)
        tokens_a = tokenizer.tokenize(tweet['data'])
        if class_to_idx:
            tokens_d = tokenizer.tokenize(tweet['dom'])
        if tweet['target'] == 'Negative':
            tweet['harm'] = '0'
        elif tweet['target'] == 'Neutral':
            tweet['harm'] = '1'
        elif tweet['target'] == 'Positive':
            tweet['harm'] = '2'

        def bert_preprocess(tokens_a, max_len, max_seq_length):
            if len(tokens_a) > max_seq_length - 2:
                tokens_a = tokens_a[:max_seq_length - 2]

            tokens = ["[CLS]"] + tokens_a + ["[SEP]"]
            segment_ids = [0] * len(tokens)

            input_ids = tokenizer.convert_tokens_to_ids(tokens)

            if len(input_ids) > max_len:
                max_len = len(input_ids)

            input_mask = [1] * len(input_ids)

            padding = [0] * (max_seq_length - len(input_ids))

            input_ids += padding
            input_mask += padding
            segment_ids += padding

            assert len(input_ids) == max_seq_length
            assert len(input_mask) == max_seq_length
            assert len(segment_ids) == max_seq_length

            return input_ids, input_mask, segment_ids

        input_ids, input_mask, segment_ids = bert_preprocess(tokens_a, max_len, max_seq_length)
        Input_ids.append(input_ids)
        Input_mask.append(input_mask)
        Segment_ids.append(segment_ids)
        if class_to_idx:
            input_ids, input_mask, segment_ids = bert_preprocess(tokens_d, max_len, max_seq_length)
            Input_ids_dom.append(input_ids)
            Input_mask_dom.append(input_mask)
            Segment_ids_dom.append(segment_ids)

        raw_data.append(tweet['data'])
        # ids.append(tweet['id'])
        # users.append(tweet['user'])
        harm.append(tweet['harm'])
        target.append(tweet['target'])
        if class_to_idx:
            dom.append(tweet['dom'])
            dom_lab.append(class_to_idx[tweet['dom']])
        # activation.append(tweet['activation'])
        # polarity.append(tweet['polarity'])
        # ment.append(mention)
        # tag.append(hash_tag)

        continue

    features['raw_data'] = np.array(raw_data)
    features['harm'] = np.array(harm)
    features['target'] = np.array(target)
    if class_to_idx:
        features['dom_raw'] = np.array(dom)
        features['dom_lab'] = np.array(dom_lab)
    # features['activation'] = np.array(activation)
    # features['polarity'] = np.array(polarity)
    # features['id'] = np.array(ids)
    # features['user'] = np.array(users)
    Input_ids = np.expand_dims(Input_ids, 1)
    Input_mask = np.expand_dims(Input_mask, 1)
    Segment_ids = np.expand_dims(Segment_ids, 1)
    features['text'] = np.concatenate((Input_ids, Input_mask, Segment_ids), axis=1)
    if class_to_idx:
        Input_ids_dom = np.expand_dims(Input_ids_dom, 1)
        Input_mask_dom = np.expand_dims(Input_mask_dom, 1)
        Segment_ids_dom = np.expand_dims(Segment_ids_dom, 1)
        features['dom'] = np.concatenate((Input_ids_dom, Input_mask_dom, Segment_ids_dom), axis=1)

    return features

def save_data(args, features, file):
    path = os.getcwd()
    with open(os.path.join(path, args.save_path + '/' + file +'.pkl'),'wb') as file_obj:
        pickle.dump(features,file_obj)
        print('Save Successful!')

if __name__ == "__main__":
    args = parse_args()
    print(args)
    class_to_idx = {}
    for file in ['train', 'dev', 'test']:
        df = pd.read_csv("twitter_data/"+file+".tsv", sep="\t", header=None)
        df.columns = ['data', 'target', 'dom']
        data = df
        if file == 'train':
            classes = data['dom'].unique().tolist()
            idx_to_class = {i: j for i, j in enumerate(classes)}
            class_to_idx = {value: key for key, value in idx_to_class.items()}
            print('class_to_idx:', class_to_idx)
        bert_Feature = bert_feature(args, data, class_to_idx)
        save_data(args, bert_Feature, file)


from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtWidgets import ( QWidget)

class BottomToolbar(object):

    def __init__(self,parent):
        self.parent = parent

    def Mainpage (self, Widget):
        self.tollbar1 = QtWidgets.QWidget(Widget)
        self.tollbar1.setGeometry(QtCore.QRect(31, 470, 981, 300))
        self.tollbar1.setObjectName("tollbar1")
        self.vertikaltollbar1Layout = QtWidgets.QVBoxLayout(self.tollbar1)
        self.vertikaltollbar1Layout.setObjectName("vertikaltollbar1Layout")
        self.horizontaltollbar1Layout = QtWidgets.QHBoxLayout()
        self.horizontaltollbar1Layout.setObjectName("horizontaltollbar1Layout")
        self.label1 = QtWidgets.QLabel()
        self.label1.setObjectName("label1")
        self.horizontaltollbar1Layout.addWidget(self.label1)
        self.lineEdit1 = QtWidgets.QLineEdit("80%")
        self.horizontaltollbar1Layout.addWidget(self.lineEdit1)
        spacerItem1 = QtWidgets.QSpacerItem(80, 20, QtWidgets.QSizePolicy.Expanding, QtWidgets.QSizePolicy.Minimum)
        self.horizontaltollbar1Layout.addItem(spacerItem1)
        self.label2 = QtWidgets.QLabel()
        self.label2.setObjectName("label2")
        self.horizontaltollbar1Layout.addWidget(self.label2)
        self.lineEdit2 = QtWidgets.QLineEdit("80%")
        self.horizontaltollbar1Layout.addWidget(self.lineEdit2)
        spacerItem2 = QtWidgets.QSpacerItem(80, 20, QtWidgets.QSizePolicy.Expanding, QtWidgets.QSizePolicy.Minimum)
        self.horizontaltollbar1Layout.addItem(spacerItem2)
        self.label3 = QtWidgets.QLabel()
        self.label3.setObjectName("label3")
        self.horizontaltollbar1Layout.addWidget(self.label3)
        self.lineEdit3 = QtWidgets.QLineEdit("80%")
        self.horizontaltollbar1Layout.addWidget(self.lineEdit3)
        spacerItem3 = QtWidgets.QSpacerItem(80, 20, QtWidgets.QSizePolicy.Expanding, QtWidgets.QSizePolicy.Minimum)
        self.horizontaltollbar1Layout.addItem(spacerItem3)
        self.vertikaltollbar1Layout.addLayout(self.horizontaltollbar1Layout)

        self.horizontaltollbar1Layout2 = QtWidgets.QHBoxLayout()
        self.horizontaltollbar1Layout2.setObjectName("horizontaltollbar1Layout2")
        spacerItem1 = QtWidgets.QSpacerItem(515, 20, QtWidgets.QSizePolicy.Expanding, QtWidgets.QSizePolicy.Minimum)
        self.horizontaltollbar1Layout2.addItem(spacerItem1)
        self.buttonProcess = QtWidgets.QPushButton()
        self.horizontaltollbar1Layout2.addWidget(self.buttonProcess)
        self.vertikaltollbar1Layout.addLayout(self.horizontaltollbar1Layout2)
        self.tollbar1.setLayout(self.vertikaltollbar1Layout)
        self.retranslateUi(self.parent)

    def retranslateUi(self, MainWindow):
        _translate = QtCore.QCoreApplication.translate
        self.label1.setText(_translate("MainWindow", "Mean (Rata - Rata)"))
        self.label2.setText(_translate("MainWindow", "Rata - rata Macro"))
        self.label3.setText(_translate("MainWindow", "Rata - rata Micro"))
        self.buttonProcess.setText(_translate("MainWindow", "Proses"))


from .preprocessing import (
    remove_links,
    remove_punct,
    remove_html_tags,
    remove_html_tags_string,
    remove_non_english,
    remove_mentions,
    remove_mentions_string,
    remove_hashtags,
    remove_hashtags_string,
    remove_emoji,
)

def preprocess_messages(df, column_name):
    """Returns a preprocessed dataframe.

    Args:
        df (dataframe): messages dataframe.
        column_name (str): the name of the text column which will be preprocessed.

    Returns:
        df (datafarme): a preprocessed dataframe.
                        Preprocessing steps: lowercase the text, remove html links, remove mentions (@...), remove hashtags,
                        remove emojis, remove punctuation.
    """

    # Remove short messages
    wantedRows = df[df[column_name].str.split().str.len() < 6].index
    df = df.drop(wantedRows, axis=0)
    df = df.reset_index(drop=True)

    # Remove empty, duplicates
    df = df.dropna()
    df = df.drop_duplicates()
    df = df.reset_index(drop=True)

    # Remove non-english messages
    df = remove_non_english(df, column_name)

    # Lowercase resources
    df[column_name] = df[column_name].str.lower()

    # Remove links
    for index, row in df.iterrows():
        sentence = df.iloc[index][column_name]
        sentence_without_links = remove_links(sentence)
        df.at[index, column_name] = sentence_without_links

    # Remove html tags
    df[column_name] = remove_html_tags(df.content)

    # Remove mentions
    df[column_name] = remove_mentions(df.content)

    # Remove hashtags
    df[column_name] = remove_hashtags(df.content)

    # Remove emojis
    for index, row in df.iterrows():
        sentence = df.iloc[index][column_name]
        sentence_without_emoji = remove_emoji(sentence)
        df.at[index, column_name] = sentence_without_emoji

    # Remove punctuation
    for index, row in df.iterrows():
        sentence = df.iloc[index][column_name]
        sentence_without_punctuation = remove_punct(sentence)
        df.at[index, column_name] = sentence_without_punctuation

    filter = df[column_name] != ""
    dfNew = df[filter]
    dfNew = dfNew.drop_duplicates().reset_index(drop=True)

    return dfNew

def preprocess_message(message):
    """Returns a preprocessed message text in a string format.

    Args:
        article (str): the message text in a string format.

    Returns:
        article (str): the preprocessed message in a string format.
                       Preprocessing steps: lowercase the text, remove html links, remove mentions (@...), remove hashtags,
                       remove emojis, remove punctuation.
    """
    message = message.lower()
    message = remove_links(message)
    message = remove_html_tags_string(message)
    message = remove_mentions_string(message)
    message = remove_hashtags_string(message)
    message = remove_emoji(message)
    message = remove_punct(message)
    return message


#!/usr/bin/python
import sys, argparse, subprocess, time, os, re, glob, numpy as np
from ftplib import FTP

# Globals
path_to_data = ''
path_to_ref = ''
path_to_results = os.path.dirname(os.path.abspath(__file__))
index_genome = False
verbosity = False
threads = 1
deleteResultFiles = False

def create_directory(mypath, verbose=False):
    try:
        if not os.path.exists(mypath):
            os.makedirs(mypath)
            if verbose:
                print("Directory created: ", mypath)
        else:
            if verbose:
                print("Directory already exists: ", mypath)

    except OSError as exc:  # Guard against race condition
        if exc.errno != errno.EEXIST:
            print("Directory could not be created: ", path)
            raise

def run_command(cmd, verbose=False, dry_run=False):
    if verbose:
        print(f'Executing command: {cmd}...', end='', flush=True)

    if dry_run:
        if verbose:
            print('Dry run: nothing executed.', end='')
    else:
        # https://docs.python.org/3/library/subprocess.html
        if type(cmd) == str:
            result = subprocess.run(cmd.split(' '), stdout=subprocess.PIPE).stdout.decode('utf-8').strip()
        elif type(cmd) == list:
            result = subprocess.run(cmd, stdout=subprocess.PIPE).stdout.decode('utf-8').strip()
        else:
            print(f"Invalid type: {type(cmd)}")

    if verbose:
        print('done.', flush=True)
        print("Result is: ", result)
    return result

def run_command_out_to_file(cmd, outfile, verbose=False, dry_run=False):
    if verbose:
        print(f'Executing command: {cmd} > {outfile}', end='', flush=True)

    if dry_run:
        if verbose:
            print('Dry run: nothing executed.', end='')
    else:
        # https://docs.python.org/3/library/subprocess.html
        f = open(outfile, "w")
        if type(cmd) == str:
            result = subprocess.run(cmd.split(' '), stdout=f)
        elif type(cmd) == list:
            result = subprocess.run(cmd, stdout=f)
        else:
            print(f"Invalid type: {type(cmd)}")
    if verbose:
        print('done.', flush=True)
        print("Result is: ", result)
    return result

def run_2pipedcommands(cmd, cmd2, verbose=False, dry_run=False):
    if verbose:
        print(f'Executing command: {cmd} | {cmd2} ...', end='', flush=True)

    if dry_run:
        if verbose:
            print('Dry run: nothing executed.', end='')
    else:
        # result = subprocess.run(cmd.split(' '), stdout=subprocess.PIPE)
        p1 = subprocess.Popen(cmd.split(' '), stdout=subprocess.PIPE)
        p2 = subprocess.Popen(cmd2.split(' '), stdin=p1.stdout, stdout=subprocess.PIPE)
        result = p2.communicate()[0]

    if verbose:
        print('done.', flush=True)
        print("Result is: ", result)
    return result

def find_mean(file):
    d = []
    fh = open(file, 'r')
    for line in fh:
        line = line.strip()
        d.append(int(line.split('\t')[2]))
    result = round(np.mean(d),3)
    print ('Mean coverage = ', result)
    return result

if __name__ == "__main__":

    parser = argparse.ArgumentParser()
    parser.add_argument("path_to_data", help="Absolute path to FASTQ data (data/subject/files)")
    parser.add_argument("path_to_ref", help="Absolute path for reference genome")
    parser.add_argument("--path_to_results", help="Absolute path for results (default is current directory)")
    parser.add_argument("--path_to_resources", help="Absolute path for GATK resources dir)")
    parser.add_argument("--variant_caller", help="Variant caller can be either haplotypecaller or deepvariant")
    parser.add_argument("--filter_vcf", help="VQSR, CNN, or NONE")
    parser.add_argument("--index_genome", help="Index the genome? (True/False)", action='store_true')
    parser.add_argument("--verbose", help="Print debugging information", action='store_true')
    parser.add_argument("--threads", help="Number of threads")
    parser.add_argument("--deleteResultFiles", help="Delete results directory after execution", action='store_true')

    args = parser.parse_args()
    if args.path_to_data:
        path_to_data = args.path_to_data
    if args.path_to_ref:
        path_to_ref = args.path_to_ref
    if args.path_to_resources:
        path_to_resources=args.path_to_resources
    if args.path_to_results:
        path_to_results = args.path_to_results
    if args.index_genome:
        index_genome = args.index_genome
    if args.verbose:
        verbosity = args.verbose
    if args.threads:
        threads = int(args.threads)
    if args.deleteResultFiles:
        deleteResultFiles = args.deleteResultFiles
    if args.variant_caller:
        var_caller = args.variant_caller
    if args.filter_vcf:
        filter=args.filter_vcf

    print('Data path is: ', path_to_data)
    print('Ref  path  is: ', path_to_ref)
    print('Resources path is: ', path_to_resources)
    print('Index Genome option is: ', index_genome)
    print('Variant Caller is: ', var_caller)
    print('Verbosity is: ', verbosity)
    print("Number of threads is: ", threads)
    print("Filter VCF option is: ",filter)
    print("deleteResultFiles option is: ", deleteResultFiles)

    if index_genome:
        run_command(f'bwa index {path_to_ref}', verbose=verbosity)

    # Traverse FASTQ data and look for paired reads only
    start_time = time.time()
    subjects = os.listdir(path_to_data)

    # If any subjects create results directory if it doesn't exist
    if subjects:
        create_directory(path_to_results, verbose=verbosity)

    for s in subjects:
        print('sample=',s)
        fastq_files = os.listdir(os.path.join(path_to_data, s))
        #print('all_files: ',fastq_files)
        # Look for _1 FASTQ files
        p = []
        for fastq in fastq_files:
                if fastq.endswith('.R1.fastq.gz'):
                        p.append(fastq)
        files = [os.path.join(path_to_data, s, f) for f in p]
        i=1
        # if we have files create an appropriate subdirectory
        if files:
            create_directory(os.path.join(path_to_results, s), verbose=verbosity)

        # For each paired read
        for f1 in files:
            # Confirm _2 FASTQ file exists
            f2 = f1.replace("R1", "R2")
            if not os.path.exists(f2):
                print("Matching Paired read file missing: ", f2)
                continue

            # If it does, run mapping commands
            finalBAM = f1.replace("R1", "_12")
            finalBAM = finalBAM.replace(path_to_data, path_to_results)
            if "fastq.gz" in finalBAM:
                finalBAM = finalBAM.replace(".fastq.gz", ".bam")
            else:
                finalBAM = finalBAM.replace(".fastq", ".bam")

            # Create our BAM
            #read group info
            LBstr='lib'+str(i)
            #LB=library# which should be unique for each because they are PCR free
            seq = f1.split('/')[-1]
            base = os.path.basename(f1)
            IDstr=base.split('.')[0].split('_')[2]+'.'+seq.split('.')[0].split('_')[3] #flowcell.lane
            PUstr=IDstr+'.'+base.split('.')[0].split('_')[1].replace('-','_') #flowcell.lane.barcode
            rg_string = f'@RG\\tID:{IDstr}\\tLB:{LBstr}\\tPL:LLUMINA\\tPU:{PUstr}\\tSM:{s}'
            print('read group string=',rg_string)
            bwamem = ['bwa', 'mem', '-Y', '-K', '10000000', '-t', str(threads), '-R', rg_string,
                      path_to_ref, f1, f2]
            run_command_out_to_file(bwamem, "out.sam", verbose=verbosity)

            # Sort Sam File
            run_command(f'gatk SortSam -I out.sam -O {finalBAM} -SO coordinate', verbose=verbosity)
            i = i+1

        # Combine all BAMs for this subject 's'
        combinedBAM = os.path.join(path_to_results, s, s + '.bam')
        combinedVCF = os.path.join(path_to_results, s, s + '_' + var_caller + '.vcf')
        filtVCF = os.path.join(path_to_results, s, s + '_' + var_caller + '.filt.vcf')
        allBAMs = glob.glob(os.path.join(path_to_results, s, '*.bam'))
        BAMarguments = ' '.join([f'I={x}' for x in allBAMs])
        dupBAM = os.path.join(path_to_results, s, s + '.markdup.bam')
        indexBAM = os.path.join(path_to_results, s, s + '.markdup.bai')
        GIAB_VCF = os.path.join(path_to_resources, 'NA12878-NISTv2.19v2_sorted.vcf')
        GIAB_BED = os.path.join(path_to_resources, 'NA12878-NISTv2.19.sorted.bed')
        known_sites = os.path.join(path_to_resources, 'Homo_sapiens_assembly38.known_indels.vcf.gz')
        training_data1 = os.path.join(path_to_resources, '1000G_omni2.5.hg38.vcf.gz')
        training_data2 = os.path.join(path_to_resources, 'hapmap_3.3.hg38.vcf.gz')

        # Mark Duplicates - use MULTIPLE INPUTS if multiple files present
        run_command(f'gatk MarkDuplicates {BAMarguments} O={dupBAM} M=metrics.txt',
                    verbose=verbosity)

        if var_caller == 'haplotypecaller':
            # BQSR Report
            run_command(
                f'gatk BaseRecalibrator -I {dupBAM} -O bqsr_out.txt --known-sites {known_sites} -R {path_to_ref}',
                verbose=verbosity)

            # Rescale bam file
            run_command(
                f'gatk ApplyBQSR -R {path_to_ref} -I {dupBAM} -O {combinedBAM} -bqsr-recal-file bqsr_out.txt',verbose=verbosity)

            # Run haplotype caller
            run_command_out_to_file(f'gatk HaplotypeCaller -I {combinedBAM} -O {combinedVCF} -R {path_to_ref} --native-pair-hmm-threads {str(threads)} --output-mode EMIT_VARIANTS_ONLY','out.vcf',verbose=verbosity)

            # See how many reads were successfully mapped
            run_command(f'samtools flagstat {combinedBAM}', verbose=verbosity)

            # Calculate mapping read depth
            run_command_out_to_file(f'samtools depth {combinedBAM}', "depth.out", verbose=verbosity)
            find_mean("depth.out")

            # Delete output files
            if deleteResultFiles:
                run_command(f'rm -rf {path_to_results}')
            s
            if filter == 'VQSR':
                # Run VQSR SNPs
                run_command(f'gatk VariantRecalibrator -V {combinedVCF} --trust-all-polymorphic -tranche 100.0 -tranche 99.95 -tranche 99.9 -tranche 99.8 -tranche 99.6 -tranche 99.5 -tranche 99.4 -tranche 99.3 -tranche 99.0 -tranche 98.0 -tranche 97.0 -tranche 90.0 -an QD -an FS -an MQ -an SOR -an DP -mode SNP -resource:hapmap,known=false,training=true,truth=true,prior=15 resources/hapmap_3.3.hg38.vcf.gz -resource:omni,known=false,training=true,truth=true,prior=12 resources/1000G_omni2.5.hg38.vcf.gz -resource:1000G,known=false,training=true,truth=false,prior=10 resources/1000G_phase1.snps.high_confidence.hg38.vcf.gz -resource:dbsnp,known=true,training=false,truth=false,prior=7 resources/Homo_sapiens_assembly38.dbsnp138.vcf -O snps.recal --tranches-file snps.tranches',verbose=verbosity)

                # Run VQSR indels
                run_command(f'gatk VariantRecalibrator -V {combinedVCF} -tranche 100.0 -tranche 99.95 -tranche 99.9 -tranche 99.5 -tranche 99.0 -tranche 97.0 -tranche 96.0 -tranche 95.0 -tranche 94.0 -tranche 93.5 -tranche 93.0 -tranche 92.0 -tranche 91.0 -tranche 90.0 -mode INDEL -resource:mills,known=false,training=true,truth=true,prior=12 resources/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz -O indels.recal --tranches-file indels.tranches -an FS -an ReadPosRankSum -an MQRankSum -an QD -an SOR -an DP',verbose=verbosity)

                # Apply VQSR
                run_command(f'gatk ApplyVQSR -R {path_to_ref} -V {combinedVCF} -O {filtVCF} --tranches-file snps.tranches --recal-file snps.recal -mode SNP', verbose=verbosity)

                #print ('Number of SNPs in final vcf file after VQSR:  ')
                run_command(f'vcftools --vcf {combinedVCF}', verbose=verbosity)

            elif filter == 'NONE':
                pass

            elif filter == 'CNN':
                print('This option is not ready yet, please use VQSR or NONE')
                    #can also add gatk CNN filter model but it needs some more troubleshooting
                #https://github.com/gatk-workflows/gatk4-cnn-variant-filter
                #https://gatk.broadinstitute.org/hc/en-us/articles/360035530952

        # Run Deepvariant
        elif var_caller == 'deepvariant':
            # Index BAM file
            print ('Indexing BAM')
            run_command(f'gatk BuildBamIndex -I {dupBAM} -O {indexBAM}',verbose=verbosity)

            # Run Deepvariant
            print ('Running Deepvariant')
            run_command('sh run-deepvariant.sh', verbose=verbosity)

            #print ('Number of SNPs in final vcf file after DeepVariant:  ')
            run_command(f'vcftools --vcf {combinedVCF}', verbose=verbosity)

            #Summarize BAM file
            # See how many reads were successfully mapped
            run_command(f'samtools flagstat {dupBAM}', verbose=True)

            # Calculate mapping read depth
            run_command_out_to_file(f'samtools depth {dupBAM}', "depth.out", verbose=verbosity)

            # Run function to find average of depth/read across all reads
            find_mean("depth.out")

            # Delete output files
            if deleteResultFiles:
                run_command(f'rm -rf {path_to_results}')

        else:
            print('variant_caller command accepts either haplotypecaller or deepvariant as options')
            pass

        #compare to GIAB
        #if s == 'HG001':
        #    run_command('sh run-hap.py.sh', verbose=verbosity)

elapsed_time = time.time() - start_time
print(f"Time taken was {round(elapsed_time)} seconds.")