# -*- coding: utf-8 -*-
r"""
Interfaces to R

This is the reference to the Sagemath R interface, usable from any
Sage program.

The %r interface creating an R cell in the sage
notebook is decribed in the Notebook manual.

The %R and %%R interface creating an R line or an R cell in the
Jupyter notebook are briefly decribed at the end of this page. This
documentation will be expanded and placed in the Jupyter notebook
manual when this manual exists.

The following examples try to follow "An Introduction to R" which can
be found at http://cran.r-project.org/doc/manuals/R-intro.html .

EXAMPLES:

Simple manipulations; numbers and vectors

The simplest data structure in R is the numeric vector which
consists of an ordered collection of numbers.  To create a
vector named $x$ using the R interface in Sage, you pass the
R interpreter object a list or tuple of numbers::

    sage: x = r([10.4,5.6,3.1,6.4,21.7]); x
    [1] 10.4  5.6  3.1  6.4 21.7

You can invert elements of a vector x in R by using the
invert operator or by doing 1/x::

    sage: ~x
    [1] 0.09615385 0.17857143 0.32258065 0.15625000 0.04608295
    sage: 1/x
    [1] 0.09615385 0.17857143 0.32258065 0.15625000 0.04608295

The following assignment creates a vector $y$ with 11 entries which
consists of two copies of $x$ with a 0 in between::

    sage: y = r([x,0,x]); y
    [1] 10.4  5.6  3.1  6.4 21.7  0.0 10.4  5.6  3.1  6.4 21.7

Vector Arithmetic

The following command generates a new vector $v$ of length 11 constructed
by adding together (element by element) $2x$ repeated 2.2 times, $y$
repeated just once, and 1 repeated 11 times::

    sage: v = 2*x+y+1; v
    [1] 32.2 17.8 10.3 20.2 66.1 21.8 22.6 12.8 16.9 50.8 43.5

One can compute the sum of the elements of an R vector in the following
two ways::

    sage: sum(x)
    [1] 47.2
    sage: x.sum()
    [1] 47.2

One can calculate the sample variance of a list of numbers::

    sage: ((x-x.mean())^2/(x.length()-1)).sum()
    [1] 53.853
    sage: x.var()
    [1] 53.853

    sage: x.sort()
    [1] 3.1  5.6  6.4 10.4 21.7
    sage: x.min()
    [1] 3.1
    sage: x.max()
    [1] 21.7
    sage: x
    [1] 10.4  5.6  3.1  6.4 21.7

    sage: r(-17).sqrt()
    [1] NaN
    sage: r('-17+0i').sqrt()
    [1] 0+4.123106i

Generating an arithmetic sequence::

    sage: r('1:10')
    [1] 1  2  3  4  5  6  7  8  9 10

Because ``from`` is a keyword in Python, it can't be used
as a keyword argument.  Instead, ``from_`` can be passed, and
R will recognize it as the correct thing::

    sage: r.seq(length=10, from_=-1, by=.2)
    [1] -1.0 -0.8 -0.6 -0.4 -0.2  0.0  0.2  0.4  0.6  0.8

    sage: x = r([10.4,5.6,3.1,6.4,21.7])
    sage: x.rep(2)
    [1] 10.4  5.6  3.1  6.4 21.7 10.4  5.6  3.1  6.4 21.7
    sage: x.rep(times=2)
    [1] 10.4  5.6  3.1  6.4 21.7 10.4  5.6  3.1  6.4 21.7
    sage: x.rep(each=2)
    [1] 10.4 10.4  5.6  5.6  3.1  3.1  6.4  6.4 21.7 21.7

Missing Values::

    sage: na = r('NA')
    sage: z = r([1,2,3,na])
    sage: z
    [1]  1  2  3 NA
    sage: ind = r.is_na(z)
    sage: ind
    [1] FALSE FALSE FALSE  TRUE
    sage: zero = r(0)
    sage: zero / zero
    [1] NaN
    sage: inf = r('Inf')
    sage: inf-inf
    [1] NaN
    sage: r.is_na(inf)
    [1] FALSE
    sage: r.is_na(inf-inf)
    [1] TRUE
    sage: r.is_na(zero/zero)
    [1] TRUE
    sage: r.is_na(na)
    [1] TRUE
    sage: r.is_nan(inf-inf)
    [1] TRUE
    sage: r.is_nan(zero/zero)
    [1] TRUE
    sage: r.is_nan(na)
    [1] FALSE

Character Vectors::

    sage: labs = r.paste('c("X","Y")', '1:10', sep='""'); labs
    [1] "X1"  "Y2"  "X3"  "Y4"  "X5"  "Y6"  "X7"  "Y8"  "X9"  "Y10"

Index vectors; selecting and modifying subsets of a data set::

    sage: na = r('NA')
    sage: x = r([10.4,5.6,3.1,6.4,21.7,na]); x
    [1] 10.4  5.6  3.1  6.4 21.7   NA
    sage: x['!is.na(self)']
    [1] 10.4  5.6  3.1  6.4 21.7

    sage: x = r([10.4,5.6,3.1,6.4,21.7,na]); x
    [1] 10.4  5.6  3.1  6.4 21.7   NA
    sage: (x+1)['(!is.na(self)) & self>0']
    [1] 11.4  6.6  4.1  7.4 22.7
    sage: x = r([10.4,-2,3.1,-0.5,21.7,na]); x
    [1] 10.4 -2.0  3.1 -0.5 21.7   NA
    sage: (x+1)['(!is.na(self)) & self>0']
    [1] 11.4  4.1  0.5 22.7

Distributions::

    sage: r.options(width="60")
    $width
    [1] 100

    sage: rr = r.dnorm(r.seq(-3,3,0.1))
    sage: rr
     [1] 0.004431848 0.005952532 0.007915452 0.010420935
     [5] 0.013582969 0.017528300 0.022394530 0.028327038
     [9] 0.035474593 0.043983596 0.053990967 0.065615815
    [13] 0.078950158 0.094049077 0.110920835 0.129517596
    [17] 0.149727466 0.171368592 0.194186055 0.217852177
    [21] 0.241970725 0.266085250 0.289691553 0.312253933
    [25] 0.333224603 0.352065327 0.368270140 0.381387815
    [29] 0.391042694 0.396952547 0.398942280 0.396952547
    [33] 0.391042694 0.381387815 0.368270140 0.352065327
    [37] 0.333224603 0.312253933 0.289691553 0.266085250
    [41] 0.241970725 0.217852177 0.194186055 0.171368592
    [45] 0.149727466 0.129517596 0.110920835 0.094049077
    [49] 0.078950158 0.065615815 0.053990967 0.043983596
    [53] 0.035474593 0.028327038 0.022394530 0.017528300
    [57] 0.013582969 0.010420935 0.007915452 0.005952532
    [61] 0.004431848

Convert R Data Structures to Python/Sage::

    sage: rr = r.dnorm(r.seq(-3,3,0.1))
    sage: sum(rr._sage_())
    9.9772125168981...

Or you get a dictionary to be able to access all the information::

    sage: rs = r.summary(r.c(1,4,3,4,3,2,5,1))
    sage: rs
       Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
      1.000   1.750   3.000   2.875   4.000   5.000
      sage: d = rs._sage_()
      sage: d['DATA']
      [1, 1.75, 3, 2.875, 4, 5]
      sage: d['_Names']
      ['Min.', '1st Qu.', 'Median', 'Mean', '3rd Qu.', 'Max.']
      sage: d['_r_class']
      ['summaryDefault', 'table']

It is also possible to access the plotting capabilities of R
through Sage.  For more information see the documentation of
r.plot() or r.png().

THE JUPYTER NOTEBOOK INTERFACE (work in progress).

The %r interface described in the Sage notebook manual is not useful
in the Jupyter notebook : it creates a inferior R interpreter which
cannot be escaped.

The RPy2 library allows the creation of an R cell in the Jupyter
notebook analogous to the %r escape in command line or %r cell in a
Sage notebook.

The interface is loaded by a cell containing the sole code :

"%load_ext rpy2.ipython"

After execution of this code, the %R and %%R magics are available :

- %R allows the execution of a single line of R code. Data exchange is
   possible via the -i and -o options. Do "%R?" in a standalone cell
   to get the documentation.

- %%R allows the execution in R of the whole text of a cell, with
    similar options (do "%%R?" in a standalone cell for
    documentation).

A few important points must be noted :

- The R interpreter launched by this interface IS (currently)
  DIFFERENT from the R interpreter used br other r... functions.

- Data exchanged via the -i and -o options have a format DIFFERENT
  from the format used by the r... functions (RPy2 mostly uses arrays,
  and bugs the user to use the pandas Python package).

- R graphics are (beautifully) displayed in output cells, but are not
  directly importable. You have to save them as .png, .pdf or .svg
  files and import them in Sage for further use.

In its current incarnation, this interface is mostly useful to
statisticians needing Sage for a few symbolic computations but mostly
using R for applied work.

AUTHORS:

- Mike Hansen (2007-11-01)
- William Stein (2008-04-19)
- Harald Schilly (2008-03-20)
- Mike Hansen (2008-04-19)
- Emmanuel Charpentier (2015-12-12, RPy2 interface)
"""

##########################################################################
#
#       Copyright (C) 2007 William Stein <wstein@gmail.com>
#                     2007 Mike Hansen   <mhansen@gmail.com>
#                     2008 Harald Schilly <harald.schilly@gmail.com>
#
#  Distributed under the terms of the GNU General Public License (GPL)
#
#                  http://www.gnu.org/licenses/
#
##########################################################################
from __future__ import print_function, absolute_import
from six.moves import range
import six

from .expect import Expect, ExpectElement, ExpectFunction, FunctionElement
from sage.env import DOT_SAGE
import re
import sage.rings.integer
from sage.structure.element import parent
from sage.misc.cachefunc import cached_method
from sage.interfaces.tab_completion import ExtraTabCompletion
from sage.docs.instancedoc import instancedoc

COMMANDS_CACHE = '%s/r_commandlist.sobj'%DOT_SAGE
PROMPT = '__SAGE__R__PROMPT__> '
prompt_re = re.compile("^>", re.M)

#there is a mirror network, but lets take #1 for now
RRepositoryURL = "http://cran.r-project.org/"
RFilteredPackages = ['.GlobalEnv']

# crosscheck with https://svn.r-project.org/R/trunk/src/main/names.c
# but package:base should cover this. i think.
RBaseCommands = ['c', "NULL", "NA", "True", "False", "Inf", "NaN"]

class R(ExtraTabCompletion, Expect):
    def __init__(self,
                 maxread=None, script_subdirectory=None,
                 server_tmpdir = None,
                 logfile=None,
                 server=None,
                 init_list_length=1024,
                 seed=None):
        """
        An interface to the R interpreter.

        R is a comprehensive collection of methods for statistics,
        modelling, bioinformatics, data analysis and much more.
        For more details, see http://www.r-project.org/about.html

        Resources:

        * http://r-project.org/ provides more information about R.
        * http://rseek.org/ R's own search engine.

        EXAMPLES::

             sage: r.summary(r.c(1,2,3,111,2,3,2,3,2,5,4))
             Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
             1.00    2.00    3.00   12.55    3.50  111.00

        TESTS::

            sage: r == loads(dumps(r))
            True
        """
        Expect.__init__(self,

                  # The capitalized version of this is used for printing.
                  name = 'r',

                  # This is regexp of the input prompt.  If you can change
                  # it to be very obfuscated that would be better.   Even
                  # better is to use sequence numbers.
                  # options(prompt=\"<prompt> \")
                  prompt = '> ', #default, later comes the change

                  # This is the command that starts up your program
                  # See #25806 for the --no-readline switch which fixes hangs for some
                  command = "R --no-readline --vanilla --quiet",

                  server=server,
                  server_tmpdir=server_tmpdir,

                  script_subdirectory = script_subdirectory,

                  # If this is true, then whenever the user presses Control-C to
                  # interrupt a calculation, the whole interface is restarted.
                  restart_on_ctrlc = False,

                  # If true, print out a message when starting
                  # up the command when you first send a command
                  # to this interface.
                  verbose_start = False,

                  logfile=logfile,

                  # If an input is longer than this number of characters, then
                  # try to switch to outputting to a file.
                  eval_using_file_cutoff=1024)

        self.__seq = 0
        self.__var_store_len = 0
        self.__init_list_length = init_list_length
        self._prompt_wait = [self._prompt]
        self._seed = seed

    def set_seed(self, seed=None):
        """
        Set the seed for R interpreter.

        The seed should be an integer.

        EXAMPLES::

            sage: r = R()
            sage: r.set_seed(1)
            1
            sage: r.sample("1:10", 5)
            [1] 3 4 5 7 2
        """
        if seed is None:
            seed = self.rand_seed()
        self.eval('set.seed(%d)' % seed)
        self._seed = seed
        return seed

    def _start(self):
        """
        Start up the R interpreter and sets the initial prompt and options.

        This is called the first time the R interface is actually used.

        EXAMPLES::

            sage: r = R()
            sage: r._start()
        """
        Expect._start(self)

        # width is line width, what's a good value? maximum is 10000!
        # pager needed to replace help view from less to printout
        # option device= is for plotting, is set to x11, NULL would be better?
        self._change_prompt(PROMPT)
        self.eval('options(prompt=\"%s\",continue=\"%s\", width=100,pager="cat",device="png")'%(PROMPT, PROMPT))
        self.expect().expect(PROMPT)
        self.eval('options(repos="%s")'%RRepositoryURL)
        self.eval('options(CRAN="%s")'%RRepositoryURL)

        # don't abort on errors, just raise them!
        # necessary for non-interactive execution
        self.eval('options(error = expression(NULL))')

        # set random seed
        self.set_seed(self._seed)

    def png(self, *args, **kwds):
        """
        Creates an R PNG device.

        This should primarily be used to save an R graphic to a custom file.  Note
        that when using this in the notebook, one must plot in the same cell that
        one creates the device.  See r.plot() documentation for more information
        about plotting via R in Sage.

        These examples won't work on the many platforms where R still gets
        built without graphics support.

        EXAMPLES::

            sage: filename = tmp_filename() + '.png'
            sage: r.png(filename='"%s"'%filename)             # optional -- rgraphics
            NULL
            sage: x = r([1,2,3])
            sage: y = r([4,5,6])
            sage: r.plot(x,y) # This saves to filename, but is not viewable from command line; optional -- rgraphics
            null device
                      1
            sage: import os; os.unlink(filename) # We remove the file for doctesting; optional -- rgraphics

        We want to make sure that we actually can view R graphics, which happens
        differently on different platforms::

            sage: s = r.eval('capabilities("png")') # Should be on Linux and Solaris
            sage: t = r.eval('capabilities("aqua")') # Should be on all supported Mac versions
            sage: "TRUE" in s+t                      # optional -- rgraphics
            True
        """
        #Check to see if R has PNG support
        s = self.eval('capabilities("png")')
        t = r.eval('capabilities("aqua")')
        if "TRUE" not in s+t:
            raise RuntimeError("R was not compiled with PNG support")

        from sage.server.support import EMBEDDED_MODE
        if EMBEDDED_MODE:
            self.setwd('"%s"'%os.path.abspath('.'))
        return RFunction(self, 'png')(*args, **kwds)

    def convert_r_list(self, l):
        r"""
        Converts an R list to a Python list.

        EXAMPLES::

            sage: s = 'c(".GlobalEnv", "package:stats", "package:graphics", "package:grDevices", \n"package:utils", "package:datasets", "package:methods", "Autoloads", \n"package:base")'
            sage: r.convert_r_list(s)
            ['.GlobalEnv',
             'package:stats',
             'package:graphics',
             'package:grDevices',
             'package:utils',
             'package:datasets',
             'package:methods',
             'Autoloads',
             'package:base']
        """
        pl = []
        l = "".join(l.split("\n"))
        l = l[2:len(l)-1]
        for l in l.split(","):
            pl.append(l.strip().strip('"'))
        return pl

    def install_packages(self, package_name):
        """
        Install an R package into Sage's R installation.

        EXAMPLES::

            sage: r.install_packages('aaMI')       # not tested
            ...
            R is free software and comes with ABSOLUTELY NO WARRANTY.
            You are welcome to redistribute it under certain conditions.
            Type 'license()' or 'licence()' for distribution details.
            ...
            Please restart Sage in order to use 'aaMI'.
        """
        cmd = """options(repos="%s"); install.packages("%s")"""%(RRepositoryURL, package_name)
        os.system("time echo '%s' | R --vanilla"%cmd)
        print("Please restart Sage in order to use '%s'." % package_name)

    def _repr_(self):
        """
        Return string representation of this R interface.

        EXAMPLES::

            sage: r                 # indirect doctest
            R Interpreter
        """
        return 'R Interpreter'

    def __reduce__(self):
        """
        Used in serializing an R interface.

        EXAMPLES::

            sage: rlr, t = r.__reduce__()
            sage: rlr(*t)
            R Interpreter
        """
        return reduce_load_R, tuple([])

    def __getattr__(self, attrname):
        """
        Called when you get an attribute of the R interface.  This
        manufactures an R function, which is a Python function that
        can then be called with various inputs.

        EXAMPLES::

            sage: c = r.c; c
            c
            sage: type(c)
            <class 'sage.interfaces.r.RFunction'>
        """
        if attrname[:1] == "_":
            raise AttributeError
        return RFunction(self, attrname)

    def _quit_string(self):
        r"""
        Return the string that when typed into R causes the R
        interpreter to exit.

        EXAMPLES::

            sage: r._quit_string()
            'quit(save="no")'
        """
        return 'quit(save="no")'

    def _read_in_file_command(self, filename):
        r"""
        Return the R command (as a string) to read in a file named
        filename into the R interpreter.

        EXAMPLES::

            sage: r._read_in_file_command('file.txt')
            'file=file("file.txt",open="r")\nsource(file)'
        """
        return 'file=file("%s",open="r")\nsource(file)'%filename

    def read(self, filename):
        r"""
        Read filename into the R interpreter by calling R's source function on a
        read-only file connection.

        EXAMPLES::

            sage: filename = tmp_filename()
            sage: f = open(filename, 'w')
            sage: _ = f.write('a <- 2+2\n')
            sage: f.close()
            sage: r.read(filename)
            sage: r.get('a')
            '[1] 4'
        """
        self.eval( self._read_in_file_command(filename) )

    def _install_hints(self):
        """
        EXAMPLES::

            sage: print(r._install_hints())
            R is currently installed with Sage.
        """
        return "R is currently installed with Sage.\n"

    def _source(self, s):
        """
        Returns the source code of an R function as a string.

        INPUT:

        - s -- the name of the function as a string

        EXAMPLES::

            sage: print(r._source("c"))
            function (...)  .Primitive("c")
        """
        if s[-2:] == "()":
            s = s[-2:]
        return self.eval('%s'%s)

    def source(self, s):
        """
        Display the R source (if possible) about the function named s.

        INPUT:

        - s -- a string representing the function whose source code you want to see

        OUTPUT: string -- source code

        EXAMPLES::

            sage: print(r.source("c"))
            function (...)  .Primitive("c")
        """
        return self._source(s)

    def version(self):
        """
        Return the version of R currently running.

        OUTPUT: tuple of ints; string

        EXAMPLES::

            sage: r.version() # not tested
            ((3, 0, 1), 'R version 3.0.1 (2013-05-16)')
            sage: rint, rstr = r.version()
            sage: rint[0] >= 3
            True
            sage: rstr.startswith('R version')
            True
        """
        major_re = re.compile(r'^major\s*(\d.*?)$', re.M)
        minor_re = re.compile(r'^minor\s*(\d.*?)$', re.M)
        version_string_re = re.compile(r'^version.string\s*(R.*?)$', re.M)

        s = self.eval('version')

        major = int( major_re.findall(s)[0].strip() )
        minor = tuple(int(i) for i in minor_re.findall(s)[0].strip().split(".") )
        version_string = version_string_re.findall(s)[0].strip()

        return ( (major,) + minor, version_string )

    def library(self, library_name):
        """
        Load the library library_name into the R interpreter.

        This function raises an ImportError if the given library
        is not known.

        INPUT:

        - library_name -- string

        EXAMPLES::

            sage: r.library('grid')
            sage: 'grid' in r.eval('(.packages())')
            True
            sage: r.library('foobar')
            Traceback (most recent call last):
            ...
            ImportError: ...
        """
        ret = self.eval('require("%s")' % library_name)
        try:
            ret = ret.decode('utf-8')
        except UnicodeDecodeError:
            ret = ret.decode('latin-1')
        # try hard to parse the message string in a locale-independent way
        if ' library(' in ret:       # locale-independent key-word
            raise ImportError("%s"%ret)
        else:
            try:
                # We need to rebuild keywords!
                del self.__tab_completion
            except AttributeError:
                pass
            self._tab_completion(verbose=False, use_disk_cache=False)

    require = library #overwrites require

    def available_packages(self):
        """
        Returns a list of all available R package names.

        This list is not necessarily sorted.

        OUTPUT: list of strings

        .. note::

            This requires an internet connection. The CRAN server is
            that is checked is defined at the top of sage/interfaces/r.py.

        EXAMPLES::

            sage: ap = r.available_packages()   # optional - internet
            sage: len(ap) > 20                  # optional - internet
            True
        """
        p = self.new('available.packages("%s/src/contrib")'%RRepositoryURL)
        s = str(p).splitlines()[1:]
        v = [x.split()[0].strip("'") for x in s]
        return v
        #The following was more structural, but breaks on my machine.  (stein)
        #p = p._sage_()
        #s = p['_Dim'][0]
        #l = [[p['DATA'][i],p['DATA'][s+1+i]] for i in range(0,s)]
        #return l

    def _object_class(self):
        """
        Return the underlying class of elements of the R interface object.

        OUTPUT: a Python class

        EXAMPLES::

            sage: r._object_class()
            <class 'sage.interfaces.r.RElement'>
        """
        return RElement

    def _true_symbol(self):
        """
        Return the symbol that represents True in R.

        OUTPUT: string

        EXAMPLES::

            sage: r._true_symbol()
            '[1] TRUE'

        This is used behinds the scenes to implement comparison::

            sage: r('1') == r('1')
            [1] TRUE
            sage: r('1') == r('2')
            [1] FALSE
        """
        # return the string rep of truth, i.e., what the system outputs
        # when you type 1==1.
        return "[1] TRUE"

    def _false_symbol(self):
        """
        Return the symbol that represents True in R.

        OUTPUT: string

        EXAMPLES::

            sage: r._false_symbol()
            '[1] FALSE'
        """
        # return the string rep of false, i.e., what the system outputs
        # when you type 1==2.
        return "[1] FALSE"

    def _equality_symbol(self):
        """
        EXAMPLES::

            sage: r._equality_symbol()
            '=='
        """
        # return the symbol for checking equality, e.g., == or eq.
        return "=="

    def help(self, command):
        """
        Returns help string for a given command.

        INPUT:
        - command -- a string

        OUTPUT: HelpExpression -- a subclass of string whose __repr__ method is __str__, so it prints nicely

        EXAMPLES::

            sage: r.help('c')
            c                     package:base                     R Documentation
            ...

            .. note::

                This is similar to typing r.command?.
        """
        s = self.eval('help("%s")'%command).strip()     # ?cmd is only an unsafe shortcut
        import sage.plot.plot
        if sage.plot.plot.EMBEDDED_MODE:
            s = s.replace('_\x08','')
        return HelpExpression(s)

    def _assign_symbol(self):
        """
        Return the symbol used in R for assignment, which is ' <- '.

        OUTPUT: string

        EXAMPLES::

            sage: r._assign_symbol()
            ' <- '
        """
        return " <- "

    def _left_list_delim(self):
        """
        Return the left delimiter for lists in R, which is 'c('

        OUTPUT: string

        EXAMPLES::

            sage: r._left_list_delim()
            'c('
        """
        return "c("

    def _right_list_delim(self):
        """
        Return the right delimiter for lists in R, which is 'c('

        OUTPUT: string

        EXAMPLES::

            sage: r._right_list_delim()
            ')'
        """
        return ")"

    def console(self):
        """
        Runs the R console as a separate new R process.

        EXAMPLES::

            sage: r.console()                    # not tested
                R version 2.6.1 (2007-11-26)
                Copyright (C) 2007 The R Foundation for Statistical Computing
                ISBN 3-900051-07-0
                ...
        """
        r_console()

    def function_call(self, function, args=None, kwds=None):
        """
        Return the result of calling an R function, with given args and keyword args.

        OUTPUT: RElement -- an object in R

        EXAMPLES::

            sage: r.function_call('length', args=[ [1,2,3] ])
            [1] 3
        """
        args, kwds = self._convert_args_kwds(args, kwds)
        self._check_valid_function_name(function)
        return self.new("%s(%s)"%(function, ",".join([s.name() for s in args] +
                                                     [self._sage_to_r_name(key)+'='+kwds[key].name() for key in kwds ] )))

    def call(self, function_name, *args, **kwds):
        r"""
        This is an alias for :meth:`function_call`.

        EXAMPLES::

            sage: r.call('length', [1,2,3])
            [1] 3
        """
        return self.function_call(function_name, args=args, kwds=kwds)

    def _an_element_impl(self):
        """
        Returns an element belonging to the R interpreter.  This is used
        behind the scenes when doing things like comparisons, etc.

        OUTPUT: RElement -- an R element.

        EXAMPLES::

            sage: r._an_element_impl()
            [1] 0
            sage: type(_)
            <class 'sage.interfaces.r.RElement'>
        """
        return self(0)

    def set(self, var, value):
        """
        Set the variable var in R to what the string value evaluates to in R.

        INPUT:

        - var -- a string
        - value -- a string

        EXAMPLES::

            sage: r.set('a', '2 + 3')
            sage: r.get('a')
            '[1] 5'
        """
        cmd = '%s <- %s'%(var,value)
        out = self.eval(cmd)
        if out.find("error") != -1:
            raise TypeError("Error executing code in R\nCODE:\n\t%s\nR ERROR:\n\t%s"%(cmd, out))

    def get(self, var):
        """
        Returns the string representation of the variable var.

        INPUT:

        - var -- a string

        OUTPUT: string

        EXAMPLES::

            sage: r.set('a', 2)
            sage: r.get('a')
            '[1] 2'
        """
        s = self.eval('%s'%var)
        #return self._remove_indices_re.sub("", s).strip()
        return s

    def na(self):
        """
        Returns the NA in R.

        OUTPUT: RElement -- an element of R

        EXAMPLES::

            sage: r.na()
            [1] NA
        """
        return self('NA')

    def completions(self, s):
        """
        Return all commands names that complete the command starting with the
        string s.   This is like typing s[Ctrl-T] in the R interpreter.

        INPUT:

        - s -- string

        OUTPUT: list -- a list of strings

        EXAMPLES::

            sage: dummy = r._tab_completion(use_disk_cache=False)    #clean doctest
            sage: r.completions('tes')
            ['testInheritedMethods', 'testPlatformEquivalence', 'testVirtual']
        """
        return [name for name in self._tab_completion() if name[:len(s)] == s]

    def _commands(self):
        """
        Return list of all commands defined in R.

        OUTPUT: list -- a sorted list of strings

        EXAMPLES::

            sage: l = r._commands()
            sage: 'AIC' in l
            True
            sage: len(l) > 200
            True
        """
        v = RBaseCommands

        ll = self.eval('dput(search())') # loaded libs
        ll = self.convert_r_list(ll)

        for lib in ll:
            if lib in RFilteredPackages:
                continue

            if lib.find("package:") != 0:
                continue #only packages

            raw = self.eval('dput(objects("%s"))'%lib)
            raw = self.convert_r_list(raw)
            raw = [x.replace(".","_") for x in raw]

            #TODO are there others? many of them are shortcuts or
            #should be done on another level, like selections in lists
            #instead of calling obj.[[( fun-args) or other crazy stuff like that

            #TODO further filtering, check if strings are now
            #really functions, in R: exists(s, mode = "function"))
            # (apply to vector with sapply(vec,func))

            #filter only python compatible identifiers
            valid = re.compile('[^a-zA-Z0-9_]+')
            raw = [x for x in raw if valid.search(x) is None]
            v += raw

        v.sort()
        return v

    def _tab_completion(self, verbose=True, use_disk_cache=True):
        """
        Return list of all R functions.

        INPUT:

        - verbose -- bool (default: True); if True, display debugging information
        - use_disk_cache -- bool (default: True); if True, use the disk cache of
          tab completions to save time.

        OUTPUT: list -- list of string

        EXAMPLES::

            sage: t = r._tab_completion(verbose=False)
            sage: len(t) > 200
            True
        """
        try:
            return self.__tab_completion
        except AttributeError:
            import sage.misc.persist
            if use_disk_cache:
                try:
                    self.__tab_completion = sage.misc.persist.load(COMMANDS_CACHE)
                    return self.__tab_completion
                except IOError:
                    pass
            if verbose and use_disk_cache:
                print("\nBuilding R command completion list (this takes")
                print("a few seconds only the first time you do it).")
                print("To force rebuild later, delete %s." % COMMANDS_CACHE)
            v = self._commands()
            self.__tab_completion = v
            if len(v) > 200 and use_disk_cache:
                sage.misc.persist.save(v, COMMANDS_CACHE)
            return v

    def plot(self, *args, **kwds):
        """
        The R plot function.  Type r.help('plot') for much more extensive
        documentation about this function.  See also below for a brief
        introduction to more plotting with R.

        If one simply wants to view an R graphic, using this function is
        is sufficient (because it calls dev.off() to turn off the device).

        However, if one wants to save the graphic to a specific file, it
        should be used as in the example below to write the output.

        EXAMPLES:

        This example saves a plot to the standard R output, usually
        a filename like ``Rplot001.png`` - from the command line, in
        the current directory, and in the cell directory in the notebook::

            sage: d=r.setwd('"%s"'%SAGE_TMP)    # for doctesting only; ignore if you are trying this
            sage: r.plot("1:10")                # optional -- rgraphics
            null device
                      1

        To save to a specific file name, one should use :meth:`png` to set
        the output device to that file.  If this is done in the notebook, it
        must be done in the same cell as the plot itself::

            sage: filename = tmp_filename() + '.png'
            sage: r.png(filename='"%s"'%filename) # Note the double quotes in single quotes!; optional -- rgraphics
            NULL
            sage: x = r([1,2,3])
            sage: y = r([4,5,6])
            sage: r.plot(x,y)         # optional -- rgraphics
            null device
                      1
            sage: import os; os.unlink(filename) # For doctesting, we remove the file; optional -- rgraphics

        Please note that for more extensive use of R's plotting
        capabilities (such as the lattices package), it is advisable
        to either use an interactive plotting device or to use the
        notebook.  The following examples are not tested, because they
        differ depending on operating system::

            sage: r.X11() # not tested - opens interactive device on systems with X11 support
            sage: r.quartz() # not tested - opens interactive device on OSX
            sage: r.hist("rnorm(100)") # not tested - makes a plot
            sage: r.library("lattice") # not tested - loads R lattice plotting package
            sage: r.histogram(x = "~ wt | cyl", data="mtcars") # not tested - makes a lattice plot
            sage: r.dev_off() # not tested, turns off the interactive viewer

        In the notebook, one can use r.png() to open the device, but
        would need to use the following since R lattice graphics do
        not automatically print away from the command line::

            sage: filename = tmp_filename() + '.png' # Not needed in notebook, used for doctesting
            sage: r.png(filename='"%s"'%filename) # filename not needed in notebook, used for doctesting; optional -- rgraphics
            NULL
            sage: r.library("lattice")
            sage: r("print(histogram(~wt | cyl, data=mtcars))") # plot should appear; optional -- rgraphics
            sage: import os; os.unlink(filename) # We remove the file for doctesting, not needed in notebook; optional -- rgraphics
        """
        # We have to define this to override the plot function defined in the
        # superclass.
        from sage.server.support import EMBEDDED_MODE
        if EMBEDDED_MODE:
            self.setwd('"%s"'%os.path.abspath('.'))
        RFunction(self, 'plot')(*args, **kwds)
        return RFunction(self, 'dev.off')()

    def _strip_prompt(self, code):
        """
        Remove the standard R prompt from the beginning of lines in code.

        INPUT:

        - code -- a string

        OUTPUT: a string

        EXAMPLES::

            sage: s = '> a <- 2\n> b <- 3'
            sage: r._strip_prompt(s)
            ' a <- 2\n b <- 3'
        """
        return prompt_re.sub("", code)

    def eval(self, code, globals=None, locals=None, synchronize=True, *args, **kwds):
        """
        Evaluates a command inside the R interpreter and returns the output
        as a string.

        EXAMPLES::

            sage: r.eval('1+1')
            '[1] 2'
        """
        # TODO split code at ";" outside of quotes and send them as individual
        #      lines without ";".
        return Expect.eval(self, code, synchronize=synchronize, *args, **kwds)

    def _r_to_sage_name(self, s):
        """
        Returns a Sage/Python identifier from an R one.  This involves
        replacing periods with underscores, <- with __, and prepending
        _ in front of Python keywords.

        INPUT:

        - s -- a string

        OUTPUT: a string

        EXAMPLES::

            sage: f = r._r_to_sage_name
            sage: f('t.test')
            't_test'
            sage: f('attr<-')
            'attr__'
            sage: f('parent.env<-')
            'parent_env__'
            sage: f('class')
            'class_'
        """
        from keyword import iskeyword
        s = s.replace('.', '_')
        s = s.replace('<-', '__')
        if iskeyword(s):
            s += '_'
        return s

    def _sage_to_r_name(self, s):
        r"""
        The reverse of :meth:`_r_to_sage_name`.  See the docs for that method.

        EXAMPLES::

            sage: f = r._sage_to_r_name
            sage: f('t_test')
            't.test'
            sage: f('attr__')
            'attr<-'
            sage: f('parent_env__')
            'parent.env<-'
            sage: r._r_to_sage_name(f('parent_env__'))
            'parent_env__'
            sage: f('class_')
            'class'
        """
        if len(s) > 1 and s[-2:] == "__":
            s = s[:-2] + '<-'
        if len(s) > 0 and s[-1] == '_':
            s = s[:-1]
        s = s.replace('_', '.')
        return s

    def __getitem__(self, s):
        """
        Returns the RFunction with name s.

        INPUT:

        - s -- a string
        OUTPUT: RFunction -- the R function that in R has name s

        EXAMPLES::

            sage: r['as.data.frame']
            as.data.frame
            sage: r['print']
            print
        """
        return RFunction(self, s, r_name=True)

    def chdir(self, dir):
        """
        Changes the working directory to ``dir``

        INPUT:

        - ``dir`` -- the directory to change to.

        EXAMPLES::

            sage: import tempfile
            sage: tmpdir = tempfile.mkdtemp()
            sage: r.chdir(tmpdir)

        Check that ``tmpdir`` and ``r.getwd()`` refer to the same
        directory.  We need to use ``realpath()`` in case ``$TMPDIR``
        (by default ``/tmp``) is a symbolic link (see :trac:`10264`).

        ::

            sage: os.path.realpath(tmpdir) == sageobj(r.getwd())  # known bug (trac #9970)
            True
        """
        self.execute('setwd(%r)' % dir)

# patterns for _sage_()
rel_re_param = re.compile(r'\s([\w\.]+)\s=')
rel_re_range = re.compile(r'([\d]+):([\d]+)')
rel_re_integer = re.compile(r'([^\d])([\d]+)L')
rel_re_terms = re.compile(r'terms\s*=\s*(.*?),')
rel_re_call = re.compile(r'call\s*=\s*(.*?)\),')

@instancedoc
class RElement(ExtraTabCompletion, ExpectElement):

    def _tab_completion(self):
        """
        Return a list of all methods of this object.

        .. note::

            Currently returns all R commands.

        EXAMPLES::

            sage: a = r([1,2,3])
            sage: t = a._tab_completion()
            sage: len(t) > 200
            True
        """
        # TODO: rewrite it, just take methods(class=class(self))
        return self.parent()._tab_completion()

    def tilde(self, x):
        """
        The tilde regression operator in R.

        EXAMPLES::

            sage: x = r([1,2,3,4,5])
            sage: y = r([3,5,7,9,11])
            sage: a = r.lm( y.tilde(x) ) # lm( y ~ x )
            sage: d = a._sage_()
            sage: d['DATA']['coefficients']['DATA'][1]
            2
        """
        par = self.parent()
        rx = par(x)
        return par.new("%s ~ %s" % (self.name(), rx.name()))

    stat_model = tilde

    def __len__(self):
        """
        Return the length of this object.

        OUTPUT: integer

        EXAMPLES::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: len(x)
            5
        """
        return int(self.parent().eval('dput(length(%s))'%self.name())[:-1] )

    def __getattr__(self, attrname):
        """
        Return attribute of this object, which is an R function with this object
        as the first input.

        INPUT:

        - attrname -- string

        OUTPUT: RFunctionElement

        EXAMPLES::

            sage: x = r([1,2,3])
            sage: length = x.length
            sage: type(length)
            <class 'sage.interfaces.r.RFunctionElement'>
            sage: length()
            [1] 3
        """
        self._check_valid()
        if attrname[:1] == "_":
            raise AttributeError
        return RFunctionElement(self, attrname)

    def __getitem__(self, n):
        """
        Return element(s) of self.

        INPUT:

        - n -- an integer, a tuple, a string that makes sense to R, or an RElement

        OUTPUT: RElement

        EXAMPLES::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: x[0]
            numeric(0)
            sage: x[1]
            [1] 10.4
            sage: x[-1]
            [1] 5.6  3.1  6.4 21.7
            sage: x[-2]
            [1] 10.4  3.1  6.4 21.7
            sage: x[-3]
            [1] 10.4  5.6  6.4 21.7
            sage: x['c(2,3)']
            [1]  5.6 3.1
            sage: key = r.c(2,3)
            sage: x[key]
            [1]  5.6 3.1
            sage: m = r.array('1:3',r.c(2,4,2))
            sage: m
            , , 1
                 [,1] [,2] [,3] [,4]
            [1,]    1    3    2    1
            [2,]    2    1    3    2
            , , 2
                 [,1] [,2] [,3] [,4]
            [1,]    3    2    1    3
            [2,]    1    3    2    1
            sage: m[1,2,2]
            [1] 2
            sage: m[1,r.c(1,2),1]
            [1] 1 3
        """
        P = self._check_valid()
        if isinstance(n, six.string_types):
            n = n.replace('self', self._name)
            return P.new('%s[%s]'%(self._name, n))
        elif parent(n) is P:  # the key is RElement itself
            return P.new('%s[%s]'%(self._name, n.name()))
        elif not isinstance(n,tuple):
            return P.new('%s[%s]'%(self._name, n))
        else:
            L = []
            for i in range(len(n)):
                if parent(n[i]) is P:
                    L.append(n[i].name())
                else:
                    L.append(str(n[i]))
            return P.new('%s[%s]'%(self._name, ','.join(L)))

    def __bool__(self):
        """
        Implements bool(self).

        .. note::

            bool(self) will only return True if self == 0 contains a FALSE in its representation.

        EXAMPLES::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: bool(x)
            True
            sage: y = r([0,0,0,0])
            sage: bool(y)
            False
            sage: bool(r(0))
            False
            sage: bool(r(1))
            True
        """
        return "FALSE" in repr(self==0)

    __nonzero__ = __bool__

    def _comparison(self, other, symbol):
        """
        Used to implement comparison of two objects.

        INPUT:

        - other -- RElement
        - symbol -- string

        OUTPUT: RElement -- output is an R element; not a bool!

        TESTS::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: x._comparison(10.4, "==")
            [1] TRUE FALSE FALSE FALSE FALSE
        """
        P = self.parent()
        other = P(other)
        return P('%s %s %s'%(self.name(), symbol, other.name()))

    def __eq__(self, other):
        """
        Equality testing term by term.

        INPUT:

        - other -- RElement

        OUTPUT: RElement -- an R element; not a bool!

        EXAMPLES:

        Notice that comparison is term by term and returns an R element. ::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: x == 10.4
            [1] TRUE FALSE FALSE FALSE FALSE
        """
        return self._comparison(other, "==")

    def __lt__(self, other):
        """
        Less than testing term by term.

        INPUT:

        - other -- RElement

        OUTPUT: RElement -- an R element; not a bool!

        EXAMPLES:

        Notice that comparison is term by term and returns an R element. ::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: x < 7
            [1] FALSE  TRUE  TRUE  TRUE FALSE
        """
        return self._comparison(other, "<")

    def __gt__(self, other):
        """
        Greater than testing term by term.

        INPUT:

        - other -- RElement

        OUTPUT: RElement -- an R element; not a bool!

        EXAMPLES:

        Notice that comparison is term by term and returns an R element. ::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: x > 8
            [1] TRUE FALSE FALSE FALSE  TRUE
        """
        return self._comparison(other, ">")

    def __le__(self, other):
        """
        Less than or equal testing term by term.

        INPUT:

        - other -- RElement

        OUTPUT: RElement -- an R element; not a bool!

        EXAMPLES::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: x <= 10.4
            [1] TRUE  TRUE  TRUE  TRUE FALSE
        """
        return self._comparison(other, "<=")

    def __ge__(self, other):
        """
        Greater than or equal testing term by term.

        INPUT:

        - other -- RElement

        OUTPUT: RElement -- an R element; not a bool!

        EXAMPLES::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: x >= 10.4
            [1] TRUE FALSE FALSE FALSE  TRUE
        """
        return self._comparison(other, ">=")

    def __ne__(self, other):
        """
        Not equal testing term by term.

        INPUT:

        - other -- RElement

        OUTPUT: RElement -- an R element; not a bool!

        EXAMPLES::

            sage: x = r([10.4,5.6,3.1,6.4,21.7])
            sage: x != 10.4
            [1] FALSE  TRUE  TRUE  TRUE  TRUE

        """
        return self._comparison(other, "!=")

    def dot_product(self, other):
        """
        Implements the notation self . other.

        INPUT:

        - self, other -- R elements

        OUTPUT: R element

        EXAMPLES::

            sage: c = r.c(1,2,3,4)
            sage: c.dot_product(c.t())
                 [,1] [,2] [,3] [,4]
            [1,]    1    2    3    4
            [2,]    2    4    6    8
            [3,]    3    6    9   12
            [4,]    4    8   12   16

            sage: v = r([3,-1,8])
            sage: v.dot_product(v)
                 [,1]
            [1,]   74
        """
        P = self._check_valid()
        Q = P(other)
        # the R operator is %*% for matrix multiplication
        return P('%s %%*%% %s'%(self.name(), Q.name()))

    def _subs_dots(self, x):
        r"""
        Replace dots by underscores; used internally to implement
        conversation from R expression to Sage objects.

        INPUT:

        - x -- regular expression match: ``<type '_sre.SRE_Match'>``

        OUTPUT: string

        EXAMPLES::

            sage: import re
            sage: a = r([1,2,3])
            sage: rel_re_param = re.compile(r'\s([\w\.]+)\s=')
            sage: rel_re_param.sub(a._subs_dots, ' test.test =')
             ' test_test ='
        """
        return x.group().replace('.','_')

    def _subs_range(self, x):
        r"""
        Change endpoints of ranges.  This is used internally in the
        code for converting R expressions to Sage objects.

        INPUT:

        - x -- regular expression match: ``<type '_sre.SRE_Match'>``

        OUTPUT: string

        EXAMPLES::

            sage: import re
            sage: a = r([1,2,3])
            sage: rel_re_range = re.compile(r'([\d]+):([\d]+)')
            sage: rel_re_range.sub(a._subs_range, ' 1:10')
            ' range(1,11)'
        """
        g = x.groups()
        g1 = int(g[1]) + 1
        return 'range(%s,%s)' % (g[0],  g1)

    def _subs_integer(self, x):
        r"""
        Replaces strings like 'dL' with 'Integer(d)' where d is some
        integer.  This is used internally in the code for converting R
        expressions to Sage objects.

        EXAMPLES::

            sage: import re
            sage: a = r([1,2,3])
            sage: rel_re_integer = re.compile(r'([^\d])([\d]+)L')
            sage: rel_re_integer.sub(a._subs_integer, ' 1L 2L')
            ' Integer(1) Integer(2)'
            sage: rel_re_integer.sub(a._subs_integer, '1L 2L')
            '1L Integer(2)'

        """
        return '%sInteger(%s)'%x.groups()

    def _convert_nested_r_list(self, exp):
        """
        Converts a string representing a (possibly) nested list in R
        to a (possibly) nested Python list.  This is used internally
        in the code for converting R expressions to Sage objects.

        INPUT:

        - exp -- a string

        OUTPUT: a string

        EXAMPLES::

            sage: a = r([1,2,3])
            sage: s = 'c(1, 2, 3)'
            sage: a._convert_nested_r_list(s)
            '[1, 2, 3]'
        """
        from re import compile as re_compile
        from re import split   as re_split
        splt = re_compile(r'(c\(|\(|\))') # c( or ( or )
        lvl = 0
        ret = []
        for token in re_split(splt, exp):
            if token == 'c(':
                ret.append('[')
                lvl += 1
            elif token == '(':
                ret.append(token)
                if lvl > 0 : lvl += 1
            elif token == ')':
                if lvl == 1:
                    ret.append(']')
                    lvl -= 1
                else:
                    ret.append(token)
                    if lvl > 0:
                        lvl -= 1
            else:
                ret.append(token)

        return ''.join(ret)

    def _r_list(self, *args, **kwds):
        """
        This is used internally in the code for converting R
        expressions to Sage objects.

        EXAMPLES::

            sage: a = r([1,2,3])
            sage: sorted(a._r_list(1,2,3,k=5).items())
            [('#0', 1), ('#1', 2), ('#2', 3), ('k', 5)]
        """
        ret = dict(kwds)
        i = 0
        for k in args:
            ret['#%s' % i] = k
            i += 1
        return ret

    def _r_structure(self, __DATA__, **kwds):
        """
        This is used internally in the code for converting R
        expressions to Sage objects.

        EXAMPLES::

            sage: a = r([1,2,3])
            sage: d = a._r_structure('data', a=1, b=2)
            sage: sorted(d.items())
            [('DATA', 'data'), ('a', 1), ('b', 2)]
            sage: a._r_structure([1,2,3,4], _Dim=(2,2))
            [1 3]
            [2 4]

        """
        if '_Dim' in kwds: #we have a matrix
            # TODO what about more than 2 dimensions?
            #      additional checks!!
            try:
                from sage.matrix.constructor import matrix
                d = kwds.get('_Dim')
                # TODO: higher dimensions? happens often in statistics
                if len(d) != 2:
                    raise TypeError
                #since R does it the other way round, we assign
                #transposed and then transpose the matrix :)
                m = matrix(d[1], d[0], [i for i in __DATA__])
                return m.transpose()
            except TypeError:
                pass
        d = dict(DATA=__DATA__)
        d.update(kwds)
        return d

    def _sage_(self):
        r"""
        Returns Sage representation of the R object.

        R objects are basic C structures, of different kind, that can
        be stacked together.  This is similar to Python lists with
        variable objects, including lists of lists.  If R lists have
        names, they are translated to a Python dictionary, with anonymous
        list entries called ``#{number}``.

        OUTPUT: object -- Python object

        EXAMPLES::

            sage: rs = r.summary(r.c(1,4,3,4,3,2,5,1))
            sage: d = rs._sage_()
            sage: sorted(d.items())
            [('DATA', [1, 1.75, 3, 2.875, 4, 5]),
             ('_Names', ['Min.', '1st Qu.', 'Median', 'Mean', '3rd Qu.', 'Max.']),
             ('_r_class', ['summaryDefault', 'table'])]
        """
        self._check_valid()
        P = self.parent()

        # This is the core of the trick: using dput
        # dput prints out the internal structure of R's data elements
        # options via .deparseOpts(control=...)
        # TODO: dput also works with a file, if things get huge!
        # [[However, using a file for output often isn't necessary
        # since pipe buffering works pretty well for output.
        # That said, benchmark this.  -- William Stein]]
        exp = P.eval('dput(%s)'%self.name())

        # Preprocess expression
        # example what this could be:
        # structure(list(statistic = structure(0.233549683248457, .Names = "t"),
        # parameter = structure(5.58461538461538, .Names = "df"), p.value = 0.823657802106985,
        # conf.int = structure(c(-2.41722062247400, 2.91722062247400
        # ), conf.level = 0.95), estimate = structure(c(2.75, 2.5), .Names = c("mean of x",
        # "mean of y")), null.value = structure(0, .Names = "difference in means"),
        # alternative = "two.sided", method = "Welch Two Sample t-test",
        # data.name = "c(1, 2, 3, 5) and c(1, 2, 3, 4)"), .Names = c("statistic",
        # "parameter", "p.value", "conf.int", "estimate", "null.value",
        # "alternative", "method", "data.name"), class = "htest")

        # R's structure (from help):
        # structure(.Data, ...)
        #    .Data: an object which will have various attributes attached to it.
        #    ...: attributes, specified in 'tag=value' form, which will be
        #         attached to data.
        #For historical reasons (these names are used when deparsing),
        # attributes '".Dim"', '".Dimnames"', '".Names"', '".Tsp"' and
        # '".Label"' are renamed to '"dim"', '"dimnames"', '"names"',
        # '"tsp"' and '"levels"'.

        # we want this in a single line
        exp.replace('\n','')
        exp = "".join(exp.split("\n"))

        # python compatible parameters
        exp = rel_re_param.sub(self._subs_dots, exp)

        # Rename class since it is a Python keyword
        exp = re.sub(' class = ', ' _r_class = ',exp)

        # Change 'structure' to '_r_structure'
        # TODO: check that we are outside of quotes ""
        exp = re.sub(r' structure\(', ' _r_structure(', exp)
        exp = re.sub(r'^structure\(', '_r_structure(', exp)  # special case

        # Change 'list' to '_r_list'
        exp = re.sub(r' list\(', ' _r_list(', exp)
        exp = re.sub(r'\(list\(', '(_r_list(', exp)

        # Change 'a:b' to 'range(a,b+1)'
        exp = rel_re_range.sub(self._subs_range, exp)

        # Change 'dL' to 'Integer(d)'
        exp = rel_re_integer.sub(self._subs_integer, exp)

        # Wrap the right hand side of terms = ... in quotes since it
        # has a ~ in it.
        exp = rel_re_terms.sub(r'terms = "\1",', exp)

        # Change call = ..., to call = "...",
        exp = rel_re_call.sub(r'call = "\1",', exp)

        # seems to work for
        # rr = r.summary(r.princomp(r.matrix(r.c(1,2,3,4,3,4,1,2,2),4)))
        # rr._sage_()
        # but the call expression get's evaluated. why?!? TODO

        # translation:
        # c is an ordered list
        # list is a dictionary (where _Names give the entries names.
        #    map entries in names to (value, name) in each entry?
        # structure is .. see above .. structure(DATA,**kw)
        # TODO: thinking of just replacing c( with ( to get a long tuple?

        exp = self._convert_nested_r_list(exp)

        # Set up the globals
        globs = {'NA':None, 'NULL':None, 'FALSE':False, 'TRUE':True,
                 '_r_list':self._r_list, '_r_structure':self._r_structure,
                 'Integer':sage.rings.integer.Integer,
                 'character':str}

        return eval(exp, globs, globs)

    def _latex_(self):
        r"""
        Return LaTeX representation of this R object.

        This calls the ``latex`` command in R.

        OUTPUT: a latex expression (basically a string)

        EXAMPLES::

            sage: latex(r(2))  # optional - Hmisc (R package)
            2
        """
        from sage.misc.latex import LatexExpr
        self._check_valid()
        P = self.parent()
        # latex is in Hmisc, this is currently not part of Sage's R!!!
        try:
            P.library('Hmisc')
        except ImportError:
            raise RuntimeError("The R package 'Hmisc' is required for R to LaTeX conversion, but it is not available.")
        return LatexExpr(P.eval('latex(%s, file="");' % self.name()))

@instancedoc
class RFunctionElement(FunctionElement):
    def __reduce__(self):
        """
        EXAMPLES::

            sage: a = r([1,2,3])
            sage: a.mean
            mean
            sage: dumps(a.mean)
            Traceback (most recent call last):
            ...
            NotImplementedError: pickling of R element methods is not yet supported
        """
        raise NotImplementedError("pickling of R element methods is not yet supported")

    def _instancedoc_(self):
        """
        Returns the help for self as a string.

        EXAMPLES::

            sage: a = r([1,2,3])
            sage: length = a.length
            sage: print(length.__doc__)
            length                 package:base                 R Documentation
            ...
            <BLANKLINE>
        """
        M = self._obj.parent()
        return M.help(self._name)

    def _sage_src_(self):
        """
        Returns the source code of self.

        EXAMPLES::

            sage: a = r([1,2,3])
            sage: length = a.length
            sage: print(length._sage_src_())
            function (x)  .Primitive("length")
        """
        M = self._obj.parent()
        return M.source(self._name)

    def __call__(self, *args, **kwds):
        """
        EXAMPLES::

            sage: a = r([1,2,3])
            sage: length = a.length
            sage: length()
            [1] 3
        """
        return self._obj.parent().function_call(self._name, args=[self._obj] + list(args), kwds=kwds)

@instancedoc
class RFunction(ExpectFunction):
    def __init__(self, parent, name, r_name=None):
        """
        A Function in the R interface.

        INPUT:

        - parent -- the R interface
        - name -- the name of the function for Python
        - r_name -- the name of the function in R itself (which can have dots in it)

        EXAMPLES::

            sage: length = r.length
            sage: type(length)
            <class 'sage.interfaces.r.RFunction'>
            sage: loads(dumps(length))
            length
        """
        self._parent = parent
        if r_name:
            self._name = name
        else:
            self._name = parent._sage_to_r_name(name)

    def __eq__(self, other):
        """
        EXAMPLES::

            sage: r.mean == loads(dumps(r.mean))
            True
            sage: r.mean == r.lr
            False
        """
        return (isinstance(other, RFunction) and
            self._name == other._name)

    def __ne__(self, other):
        """
        EXAMPLES::

            sage: r.mean != loads(dumps(r.mean))
            False
            sage: r.mean != r.lr
            True
        """
        return not (self == other)

    def _instancedoc_(self):
        """
        Returns the help for self.

        EXAMPLES::

            sage: length = r.length
            sage: print(length.__doc__)
            length                 package:base                 R Documentation
            ...
            <BLANKLINE>
        """
        M = self._parent
        return M.help(self._name)

    def _sage_src_(self):
        """
        Returns the source of self.

        EXAMPLES::

            sage: length = r.length
            sage: print(length._sage_src_())
            function (x)  .Primitive("length")

        """
        M = self._parent
        return M.source(self._name)

    def __call__(self, *args, **kwds):
        """
        EXAMPLES::

            sage: length = r.length
            sage: length([1,2,3])
            [1] 3
        """
        return self._parent.function_call(self._name, args=list(args), kwds=kwds)

def is_RElement(x):
    """
    Return True if x is an element in an R interface.

    INPUT:

    - x -- object

    OUTPUT: bool

    EXAMPLES::

        sage: from sage.interfaces.r import is_RElement
        sage: is_RElement(2)
        False
        sage: is_RElement(r(2))
        True
    """
    return isinstance(x, RElement)

# An instance of R
r = R()

def reduce_load_R():
    """
    Used for reconstructing a copy of the R interpreter from a pickle.

    EXAMPLES::

        sage: from sage.interfaces.r import reduce_load_R
        sage: reduce_load_R()
        R Interpreter
    """
    return r

import os
def r_console():
    """
    Spawn a new R command-line session.

    EXAMPLES::

        sage: r.console()                    # not tested
            R version 2.6.1 (2007-11-26)
            Copyright (C) 2007 The R Foundation for Statistical Computing
            ISBN 3-900051-07-0
            ...
    """
    from sage.repl.rich_output.display_manager import get_display_manager
    if not get_display_manager().is_in_terminal():
        raise RuntimeError('Can use the console only in the terminal. Try %%r magics instead.')
    # This will only spawn local processes
    os.system('R --vanilla')

def r_version():
    """
    Return the R version.

    EXAMPLES::

        sage: r_version() # not tested
        ((3, 0, 1), 'R version 3.0.1 (2013-05-16)')
        sage: rint, rstr = r_version()
        sage: rint[0] >= 3
        True
        sage: rstr.startswith('R version')
        True
    """
    return r.version()

class HelpExpression(str):
    """
    Used to improve printing of output of r.help.
    """
    def __repr__(self):
        """
        Return string representation of self.

        OUTPUT: string

        EXAMPLES::

            sage: a = sage.interfaces.r.HelpExpression("This\nis\nR!")
            sage: type(a)
            <class 'sage.interfaces.r.HelpExpression'>
            sage: a
            This
            is
            R!
        """
        return str(self)


import numpy as np
from scipy.linalg import solve
import random
import cv2
import paths

def base_transform(T, U_camera):
    """
    Takes camera points and transform them to the UR10 base frame.

    :param T: Transformation matrix (3x4)
    :param U: Triangulated points in camera frame (X;Y;Z)
    :return: The points in base frame
    """

    U_camera = np.vstack([U_camera, 1])
    T = np.vstack([T, [0, 0, 0, 1]])
    return np.dot(T, U_camera)[:3]

def affine_transformation(A, t, lpoints):
    """
    Calculates the coordinates of the right image using affine transformation
    rpoints=A*lpoints+t

    :param A: Transformation matrix (2x2)
    :param t: Translation vector
    :param lpoints: Points from left image
    :return: The corresponding points in the right image
    """

    mul = np.matmul(A.reshape(2, 2), lpoints)

    #rpoints = np.add(mul.reshape(2, 1), t)
    rpoints = np.add(mul, t)
    return tuple((rpoints[0], rpoints[1]))

def projective_transformation(H, lpoints):
    lpoints = np.vstack([lpoints.reshape(2, 1), 1])
    mul = np.matmul(H, lpoints)

    return tuple(mul[0], mul[1])

def projective_transformation_solver(lpoints, rpoints):
    """
    Calculates the projection transformation matrix with 4 coordinates.

    :param lpoints:  List of coordinates (x,y), of at least length 4, for left camera
    :param rpoints: List of coordinates (x,y), of at least length 4, for right camera
    :return: Transformation matrix H
    """

    M = np.matrix(
        [[lpoints[0][0], lpoints[0][1], 1, 0, 0, 0, -rpoints[0][0] * lpoints[0][0], -rpoints[0][0] * lpoints[0][1]],
         [0, 0, 0, lpoints[0][0], lpoints[0][1], 1, -rpoints[0][1] * lpoints[0][0], -rpoints[0][1] * lpoints[0][1]],
         [lpoints[1][0], lpoints[1][1], 1, 0, 0, 0, -rpoints[0][1] * lpoints[0][0], -rpoints[0][1] * lpoints[0][1]],
         [0, 0, 0, lpoints[1][0], lpoints[1][1], 1, -rpoints[0][1] * lpoints[0][0], -rpoints[0][1] * lpoints[0][1]],
         [lpoints[2][0], lpoints[2][1], 1, 0, 0, 0, -rpoints[0][1] * lpoints[0][0], -rpoints[0][1] * lpoints[0][1]],
         [0, 0, 0, lpoints[2][0], lpoints[2][1], 1, -rpoints[0][1] * lpoints[0][0], -rpoints[0][1] * lpoints[0][1]]]
    )

    v = np.matrix(
        [[rpoints[0][0]], [rpoints[0][1]], [rpoints[1][0]], [rpoints[1][1]], [rpoints[2][0]],
         [rpoints[2][1]]])
    try:
        theta = solve(M, v)
    except np.linalg.linalg.LinAlgError:
        return np.array([0, 0, 0, 0]), np.array([0, 0])

    A = np.array([theta[0], theta[1], theta[2], theta[3]])
    A = A.reshape(2, 2)
    t = np.array([theta[4], theta[5]])
    return A, t

def affine_transformation_solver(lpoints, rpoints):
    """
    Calculates the transformation matrix and translation vector with 3 coordinates.

    :param lpoints:  List of coordinates (x,y), of at least length 3, for left camera
    :param rpoints: List of coordinates (x,y), of at least length 3, for right camera
    :return: Transformation matrix A and translation vector t
    """

    M = np.matrix([[lpoints[0][0], lpoints[0][1], 0, 0, 1, 0],
                   [0, 0, lpoints[0][0], lpoints[0][1], 0, 1],
                   [lpoints[1][0], lpoints[1][1], 0, 0, 1, 0],
                   [0, 0, lpoints[1][0], lpoints[1][1], 0, 1],
                   [lpoints[2][0], lpoints[2][1], 0, 0, 1, 0],
                   [0, 0, lpoints[2][0], lpoints[2][1], 0, 1]]
                  )

    v = np.matrix(
        [[rpoints[0][0]], [rpoints[0][1]], [rpoints[1][0]], [rpoints[1][1]], [rpoints[2][0]],
         [rpoints[2][1]]])
    try:
        theta = solve(M, v)
    except np.linalg.linalg.LinAlgError:
        return np.array([0, 0, 0, 0]), np.array([0, 0])

    A = np.array([theta[0], theta[1], theta[2], theta[3]])
    A = A.reshape(2, 2)
    t = np.array([theta[4], theta[5]])
    return A, t

def get_residuals(A, t, lpoints, rpoints):
    """
    Estimates the residuals for the transformation. Used in ransac_inliers for finding good combinations of points.

    :param A: Transformation matrix
    :param t: Translation vector
    :param lpoints: Points from left image that were used for calculating A and t
    :param rpoints: Points from right image
    :return: The absolute residual
    """

    ri = np.empty((2, len(lpoints)))
    for i in range(0, len(lpoints)):
        affine = affine_transformation(A, t, lpoints[i])
        ri[0][i] = affine[0] - rpoints[i][0]
        ri[1][i] = affine[1] - rpoints[i][1]
    return np.sqrt(np.sum(np.multiply(ri, ri)))

def ransac_inliers(lpoints, rpoints, threshold):
    """
    Takes matched points from feature_match, does affine transformation and removes points which gives outliers.

    :param lpoints: Left image features from feature_matching
    :param rpoints: Right image features from feature_matching
    :param threshold: The threshold for classifying points as outliers
    :return: The inliers for left and right points
    """

    N = len(lpoints)
    new_lpoints = []
    new_rpoints = []
    loop_length = 100 + 100 * N

    for i in range(0, loop_length):
        lsample = []
        rsample = []
        index = []
        j = 0
        while j < 3:
            rand_index = random.randrange(0, len(lpoints) - 1)
            if rand_index not in index:
                lsample.append(lpoints[rand_index])
                rsample.append(rpoints[rand_index])
                index.append(rand_index)
                j += 1
        [A, t] = affine_transformation_solver(lsample, rsample)

        loss = get_residuals(A, t, lpoints, rpoints)

        if loss < threshold:
            new_lpoints.append(lsample)
            new_rpoints.append(rsample)

    unique_lpoints = []
    for tup in new_lpoints:
        for i in range(0, 3):
            if tuple(tup[i]) not in unique_lpoints:
                unique_lpoints.append(tuple(tup[i]))

    unique_rpoints = []

    for tup in new_rpoints:
        for i in range(0, 3):
            if tuple(tup[i]) not in unique_rpoints:
                unique_rpoints.append(tuple(tup[i]))

    return unique_lpoints, unique_rpoints

def least_square_solver(lpoints, rpoints, threshold):
    """
    Uses the least square method the estimate the transformation matrix and translation vector.
    Least square is sensitive to outliers therefor ransac_inliers is first used to remove them.

    :param lpoints: List of coordinates (x,y), of at least length 3, for left camera
    :param rpoints: List of coordinates (x,y), of at least length 3, for right camera
    :param threshold: The treshold used for removing outliers in RANSAC
    :return: Transformation matrix A and translation vector t
    """
    lpoints, rpoints = ransac_inliers(lpoints, rpoints, threshold)
    print('Ransac points ' + str(len(lpoints)))
    N = len(lpoints)
    if N < 3:
        print('Not enough inliers found!')
        return None, None

    M = np.array([[lpoints[0][0], lpoints[0][1], 1, 0, 0, 0], [0, 0, 0, lpoints[0][0], lpoints[0][1], 1]])
    v = np.array([[rpoints[0][0]], [rpoints[0][1]]])

    for i in range(1, N):
        M_row = np.array([[lpoints[i][0], lpoints[i][1], 1, 0, 0, 0], [0, 0, 0, lpoints[i][0], lpoints[i][1], 1]])
        v_row = np.array([[rpoints[i][0]], [rpoints[i][1]]])
        M = np.vstack((M, M_row))
        v = np.vstack((v, v_row))

    theta, r, ra, s = np.linalg.lstsq(M, v)
    A = np.array([[theta[0], theta[1]], [theta[3], theta[4]]])
    t = np.array([theta[2], theta[5]])
    return A, t

def find_image_transformation(chess=True):
    """
    Takes two chessboard images, left and right, and finds the corner of the chessboard. These are then used for
    the least_square_solver to find the transformation matrix
    Adjust the ransac threshold!
    :param chess: If the function shall create chessboard corner points
    :return: transformation matrix and translation vector
    """
    ransac__threshold = 130
    if chess:
        left_chess = cv2.imread(paths.left_chessboard)
        right_chess = cv2.imread(paths.right_chessboard)
        left_chess = cv2.cvtColor(left_chess, cv2.COLOR_BGR2GRAY)
        right_chess = cv2.cvtColor(right_chess, cv2.COLOR_BGR2GRAY)

        _, corners_left = cv2.findChessboardCorners(left_chess, (22, 16), None)
        _, corners_right = cv2.findChessboardCorners(right_chess, (22, 16), None)
        if len(corners_left) == len(corners_right):
            corners_left = [cl[0] for cl in corners_left]
            corners_right = [cr[0] for cr in corners_right]
            A, t = least_square_solver(corners_left, corners_right, ransac__threshold)
            return A, t
        else:

            print('Different amount of chessboard points')
    else:
        corners_left = np.genfromtxt('lpoints5.txt')
        corners_right = np.genfromtxt('rpoints5.txt')
        print(len(corners_left))
        print(len(corners_right))
        A, t = least_square_solver(corners_left, corners_right, ransac__threshold)
        return A, t

#print(find_image_transformation(chess=False))


hmi = 0
nhmi = ''
mmi = 0
somaI = 0
mediaI = 0
for p in range(1, 5):
    print('----- {}° Pessoa -----'.format(p))
    nome = str(input('Nome: ')).strip()
    idade = int(input('Idade: '))
    sexo = str(input('Sexo [M/F]: ')).lower().strip()
    somaI += idade
    if sexo == 'm' and idade > hmi:
        hmi = idade
        nhmi = nome
    if sexo == 'f' and idade < 20:
        mmi += 1

mediaI = somaI / 4
print('A média de idade do grupo é de {} anos'.format(mediaI))
print('O homem mais velho tem {} anos é {}'.format(hmi, nhmi))
print('Há {} mulheres menores do que 20 anos'.format(mmi))


def tick_toward(start, target):
    lst = [start]
    n = max(abs(target[0]-start[0]),abs(target[1]-start[1]))
    i = 0
    while i < n:
        if lst[i][0] < target[0]:
            num0 = lst[i][0]+1
        elif lst[i][0] > target[0]:
            num0 = lst[i][0]-1
        else:
            num0 = lst[i][0]

        if lst[i][1] < target[1]:
            num_1 = lst[i][1]+1
        elif lst[i][1] > target[1]:
            num_1 = lst[i][1]-1
        else:
            num_1 = lst[i][1]
        lst.append((num0,num_1))
        i += 1
    return lst

def tick_towardB(start, target):
    x1, y1 = target
    points = [start]
    while points[-1] != target:
        x2, y2 = points[-1]
        a, b = (x2 < x1) - (x2 > x1), (y2 < y1) - (y2 > y1)
        points.append((x2 + a, y2 + b))
    return points


import logging
import os
import requests
import stress_test.oftraf_exceptions
import sys
import traceback
import util.netutil

class Oftraf:
    """
    All Oftraf related functionality is here
    """
    def __init__(self, controller, test_config):
        """
        Creates an Oftraf Monitor Controller object.
        Options from JSON input file

        :param controller: object of the Controller class
        :param test_config: JSON input configuration
        :type controller: object
        :type test_config: parsed json file with test configuration
        """
        if 'oftraf_test_interval_ms' in test_config:
            self.interval_ms = test_config['oftraf_test_interval_ms']
        else:
            self.interval_ms = 0
        self.rest_server_port = test_config['oftraf_rest_server_port']
        self.ip = controller.ip
        self.ssh_port = controller.ssh_port
        self.ssh_user = controller.ssh_user
        self.ssh_pass = controller.ssh_pass

        self.of_port = controller.of_port
        self._ssh_conn = controller.init_ssh()
        self.traceback_enabled = False

    def _error_handling(self, error_message, error_num=1):
        """
        Handles custom errors of oftraf

        :param error_message: message of the handled error
        :param error_num: error number of the handled error, used to define
        subcases of raised errors.
        :type error_message: str
        :type error_num: int
        :raises oftraf_exceptions.OftrafError: to terminate execution of
        test after error handling
        """
        exc_type, exc_obj, exc_tb = sys.exc_info()
        logging.error('{0} :::::::::: Exception :::::::::::'.
                      format(exc_obj))
        logging.error(error_message)
        logging.error('Error number:{0}'.format(error_num))
        logging.error('{0} - {1} Exception: {2}, {3}'.
                      format(exc_obj, self.name, exc_type, exc_tb.tb_lineno))
        if self.traceback_enabled:
            traceback.print_exc()
        # Propagate error outside the class to stop execution
        raise(stress_test.oftraf_exceptions.OftrafError)

    def get_oftraf_path(self):
        """
        Returns oftraf base directory path, using as base to the project
        path

        :returns: oftraf folder path
        :rtype: str
        """
        stress_test_base_dir = os.path.abspath(os.path.join(
            os.path.realpath(__file__), os.pardir))
        monitors_base_dir = os.path.abspath(os.path.join(stress_test_base_dir,
                                                         os.pardir))
        oftraf_path = os.path.sep.join(
            [monitors_base_dir, 'monitors', 'oftraf', ''])
        return str(oftraf_path)

    def build(self):
        """
        Wrapper to the oftraf monitor build handler

        :raises IOError: if the handler does not exist on the remote host
        :raises oftraf_exceptions.OftrafBuildError: if build process fails
        """
        logging.info('[Oftraf] Building')
        try:
            try:
                oftraf_path = str(self.get_oftraf_path())
                build_hnd = os.path.join(str(oftraf_path), 'build.sh')
                if not util.netutil.isfile(self.ip,
                                           self.ssh_port,
                                           self.ssh_user,
                                           self.ssh_pass,
                                           [build_hnd]):
                    raise(IOError(
                        '{0} build handler does not exist'.
                        format('[oftraf.build]')))
                else:
                    util.netutil.make_remote_file_executable(
                        self.ip, self.ssh_port, self.ssh_user, self.ssh_pass,
                        build_hnd)
                exit_status, cmd_output = \
                    util.netutil.ssh_run_command(self._ssh_conn,
                                                 ' '.join([build_hnd]),
                                                 '[oftraf.build_handler]')
                if exit_status == 0:
                    logging.info("[OFTraf] Successful building")
                else:
                    raise(stress_test.oftraf_exceptions.OftrafBuildError(
                        'Build process exited with non zero exit code. '
                        'Command-line output: {0} \n Exit status code: {1}'.
                        format(cmd_output, exit_status), 2))
            except stress_test.oftraf_exceptions.OftrafError as e:
                self._error_handling(e.err_msg, e.err_code)
            except:
                raise(stress_test.oftraf_exceptions.OftrafBuildError)
        except stress_test.oftraf_exceptions.OftrafError as e:
            self._error_handling(e.err_msg, e.err_code)

    def clean(self):
        """
        Wrapper to the oftraf monitor clean handler

        :raises IOError: if the handler does not exist on the remote host
        :raises oftraf_exceptions.OftrafCleanError: if clean process fails
        """
        logging.info('[Oftraf] Cleaning')
        try:
            try:
                oftraf_path = self.get_oftraf_path()
                clean_hnd = oftraf_path + 'clean.sh'
                if not util.netutil.isfile(self.ip, self.ssh_port,
                                           self.ssh_user, self.ssh_pass,
                                           [clean_hnd]):
                    raise(IOError(
                        '{0} clean handler does not exist'.
                        format('[oftraf.build]')))
                else:
                    util.netutil.make_remote_file_executable(
                        self.ip, self.ssh_port, self.ssh_user, self.ssh_pass,
                        clean_hnd)
                exit_status, cmd_output = \
                    util.netutil.ssh_run_command(self._ssh_conn,
                                                 ' '.join([clean_hnd]),
                                                 '[oftraf.clean_handler]')
                if exit_status == 0:
                    logging.info("[Oftraf] Successful cleaning")
                else:
                    raise(stress_test.oftraf_exceptions.OftrafCleanError(
                        'clean process exited with non zero exit code. '
                        'Command-line output: {0} \n Exit status code: {1}'.
                        format(cmd_output, exit_status), 2))
            except stress_test.oftraf_exceptions.OftrafError as e:
                self._error_handling(e.err_msg, e.err_code)
            except:
                raise(stress_test.oftraf_exceptions.OftrafCleanError)
        except stress_test.oftraf_exceptions.OftrafError as e:
            self._error_handling(e.err_msg, e.err_code)

    def start(self):
        """
        Wrapper to the oftraf monitor start handler. Initializes the REST
        interface of oftraf and listen of traffic on controller Southbound
        interface

        :raises IOError: if the handler does not exist on the remote host
        :raises oftraf_exceptions.OftrafStartError: if start process fails
        """
        logging.info('[Oftraf] Starting')
        try:
            try:
                oftraf_path = self.get_oftraf_path()
                start_hnd = oftraf_path + 'start.sh'
                if not util.netutil.isfile(self.ip, self.ssh_port,
                                           self.ssh_user, self.ssh_pass,
                                           [start_hnd]):
                    raise(IOError(
                        '{0} start handler does not exist'.
                        format('[oftraf.build]')))
                else:
                    util.netutil.make_remote_file_executable(
                        self.ip, self.ssh_port, self.ssh_user, self.ssh_pass,
                        start_hnd)
                exit_status, cmd_output = \
                    util.netutil.ssh_run_command(
                        self._ssh_conn, ' '.join([start_hnd,
                                                  self.ip,
                                                  str(self.rest_server_port),
                                                  str(self.of_port)]),
                        '[oftraf.start_handler]',
                        lines_queue=None, print_flag=True, block_flag=True,
                        getpty_flag=True)
                if exit_status == 0:
                    logging.info("[Oftraf] Successful starting")
                else:
                    raise(stress_test.oftraf_exceptions.OftrafStartError(
                        'Start process exited with non zero exit code. '
                        'Command-line output: {0} \n Exit status code: {1}'.
                        format(cmd_output, exit_status), 2))
            except stress_test.oftraf_exceptions.OftrafError as e:
                self._error_handling(e.err_msg, e.err_code)
            except:
                raise(stress_test.oftraf_exceptions.OftrafStartError)
        except stress_test.oftraf_exceptions.OftrafError as e:
            self._error_handling(e.err_msg, e.err_code)

    def stop(self):
        """ Wrapper to the oftraf monitor stop handler

        :raises IOError: if the handler does not exist on the remote host
        :raises oftraf_exceptions.OftrafStopError: if stop process fails
        """
        logging.info('[Oftraf] Starting')
        try:
            try:
                oftraf_path = self.get_oftraf_path()
                stop_hnd = oftraf_path + 'stop.sh'
                if not util.netutil.isfile(self.ip, self.ssh_port,
                                           self.ssh_user, self.ssh_pass,
                                           [stop_hnd]):
                    raise(IOError(
                        '{0} stop handler does not exist'.
                        format('[oftraf.build]')))
                else:
                    util.netutil.make_remote_file_executable(
                        self.ip, self.ssh_port, self.ssh_user, self.ssh_pass,
                        stop_hnd)
                exit_status, cmd_output = \
                    util.netutil.ssh_run_command(
                        self._ssh_conn,
                        ' '.join([stop_hnd,
                                  self.ip,
                                  str(self.rest_server_port)]),
                        '[oftraf.stop_handler]')
                if exit_status == 0:
                    logging.info("[Oftraf] Successful stopping")
                else:
                    raise(stress_test.oftraf_exceptions.OftrafStopError(
                        'Stop process exited with non zero exit code. '
                        'Command-line output: {0} \n Exit status code: {1}'.
                        format(cmd_output, exit_status), 2))
            except stress_test.oftraf_exceptions.OftrafError as e:
                self._error_handling(e.err_msg, e.err_code)
            except:
                raise(stress_test.oftraf_exceptions.OftrafStopError)
        except stress_test.oftraf_exceptions.OftrafError as e:
            self._error_handling(e.err_msg, e.err_code)

    def oftraf_get_of_counts(self):
        """
        Gets the openFlow packets counts, measured by oftraf. It uses the
        oftraf REST interface and returns the result as a string in JSON format

        :returns: oftraf metrics as string in JSON format
        :rtype: str
        :raises oftraf_exceptions.OftrafError: if execution of handler fails
        """
        try:
            try:
                getheaders = {'Accept': 'application/json'}
                url = \
                    'http://{0}:{1}/get_of_counts'.format(
                        self.ip, self.rest_server_port)
                s = requests.Session()
                s.trust_env = False
                req = s.get(url, headers=getheaders, stream=False)
                return req.content.decode('utf-8')
            except:
                raise(stress_test.oftraf_exceptions.OftrafGetResultError(
                    'Fail getting total number of installed flows \n Oftraf '
                    'REST request status code: {0} \n Oftraf REST request '
                    'data: {1}'.format(req.status_code,
                                       req.content.decode('utf-8'))))
        except stress_test.oftraf_exceptions.OftrafError as e:
            self._error_handling(e.err_msg, e.err_code)

    def __del__(self):
        """
        Method called when object is destroyed
        """
        try:
            logging.info('Run oftraf stop.')
            self.stop()
        except Exception as e:
            logging.info('Fail stopping oftraf during cleanup. '
                         'Exception message: {0}'.format(e))

        try:
            logging.info('Run oftraf cleanup.')
            self.clean()
        except Exception as e:
            logging.info('Fail cleaning oftraf files during cleanup. '
                         'Exception message: {0}'.format(e))

        try:
            logging.info('Close oftraf node ssh connection.')
            self._ssh_conn.close()
        except Exception as e:
            logging.info('Fail closing ssh oftraf node connection during '
                         'cleanup. Exception message: {0}'.format(e))


from flask import render_template, flash, redirect, url_for, request
from flask_login import login_user, logout_user
from app import app, db, lm
from app.models.tables import User, Perfil
from app.models.forms import LoginForm, AlunoForm, PerfilForm

@lm.user_loader
def load_user(id):
    return User.query.filter_by(id=id).first()

@app.route("/index")
@app.route("/")
def index():
    return render_template('index.html')

@app.route("/login", methods=["GET", "POST"])
def login():
    form = LoginForm()
    if form.validate_on_submit():
        user = User.query.filter_by(username=form.username.data).first()
        if user and user.password == form.password.data:
            login_user(user)
            flash("Olá " + user.name + ", login realizado com sucesso!")
            if (user.password == 'coordenacao2019'):
                return redirect(url_for("coordenacao"))
            else:
                return redirect(url_for("aluno"))
        else:
            flash("Algum campo é inválido!")
    return render_template('login.html', form=form)

@app.route("/logout")
def logout():
    logout_user()
    flash("Você foi desconectado!")
    return redirect(url_for("index"))

@app.route("/coordenacao")
def coordenacao():
    return render_template('coordenacao.html')

@app.route("/aluno")
def aluno():
    return render_template('aluno.html')

@app.route("/coordenacao/listagem", methods=['GET', 'POST'])
def listagem():
    data = {}
    data['users'] = User.query.all()
    return render_template('listagem.html', data=data)

@app.route("/create/<info>")
@app.route("/create", defaults={"info": None})
def create(info):
    i = User("coordenador10", "coordenacao2019", "Coordenador10", "coordenacao10@gmail.com")
    db.session.add(i)
    db.session.commit()
    return "OK"

@app.route("/read/<info>")
@app.route("/read", defaults={"info": None})
def read(info):
    r = Perfil.query.filter_by(id=15).first()
    print(r.curso)
    return "OK"

@app.route("/update/<info>")
@app.route("/update", defaults={"info": None})
def update(info):
    r = User.query.filter_by(username="teste46").first()
    r.name = "testedeucerto"
    db.session.add(r)
    db.session.commit()
    return "OK"

@app.route("/delete/<info>")
@app.route("/delete", defaults={"info": None})
def delete(info):
    r = User.query.filter_by(username="teste46").first()
    r.name = "testedeucerto"
    db.session.delete(r)
    db.session.commit()
    return "OK"

@app.route("/coordenacao/cadaluno/", methods=['GET', 'POST'])
def cadaluno():
    form = AlunoForm()
    i = User(form.username.data, form.password.data, form.name.data, form.email.data)
    if form.validate_on_submit():
        db.session.add(i)
        db.session.commit()
        flash('Usuário Cadastrado')
    return render_template('cadaluno.html', form=form)

@app.route("/coordenacao/cadperfil/", methods=['GET', 'POST'])
def cadperfil():
    form = PerfilForm()
    i = Perfil(form.user_id.data, form.curso.data, form.periodo.data, form.disc_1.data, form.nota_d1.data, form.disc_2.data, form.nota_d2.data,
                                                    form.disc_3.data, form.nota_d3.data, form.disc_4.data, form.nota_d4.data)
    if form.validate_on_submit():
        db.session.add(i)
        db.session.commit()
        flash('Perfil Cadastrado')
    return render_template('cadperfil.html', form=form)


"""
実験場所でsshが使用できなかった場合の緊急用コード
oyoneを直進モードだけで動作させる
"""
import serial_communication
import save_sensor_img
import cv2

cap = cv2.VideoCapture(0)

def straight_only():
    to_arduino = serial_communication.communicator()
    data_saver = save_sensor_img.Data_Saver(save_video=True,
    save_sensor=True)
    while True:
        ret, frame = cap.read()

        if ret == False:
            print('Camera is not detected.')
            break
        else:
            cv2.imshow('input image',frame)
            key = cv2.waitKey(1)
            sensor_dict = None # 後でセンサー値をaruduinoから取得する関数を作成する
            which = "f"

            data_saver.save(frame, sensor_dict, which)
            print(which)
            to_arduino.serial_wirte(which)

            rtn_which = to_arduino.serial_read()

            # key27: esc
            if key == 27:
                to_arduino.serial_wirte("s")
                break

straight_only()


import unittest
from bs4 import BeautifulSoup
from web_scraping.parse import get_final_list
"""This test checks if parsing the BS object returns expected results"""

class GetFinalList(unittest.TestCase):
    def test_final_list(self):
        with open("test2_input.html", "r", encoding="utf-8") as f:
            contents = f.read()
        page_content = BeautifulSoup(contents, "html.parser")
        data_ids_list = ['110925072']
        result = get_final_list(data_ids_list, page_content)
        self.assertEqual(result, [{'id': '110925072', 'address': 'Heinrich-Roller-Straße 9, Prenzlauer Berg',
                                   'price': '790,40 €', 'living_space': '49,4 m²'}])

if __name__ == '__main__':
    unittest.main()


import tkinter as tk
import add_data, view_data

container = tk.Tk()

width = container.winfo_screenwidth()  # width of screen
height = container.winfo_screenheight()  # height of screen
container.winfo_toplevel().geometry("%dx%d%+d%+d" % (width, height, 0, 0))

start_frame = tk.Frame(container, width=width, height=height)
start_frame.pack()

tk.Label(start_frame, text="Welcome to Marks Butler", font=("Courier", 19)).place(relx=.5, rely=.3, anchor=tk.CENTER)

def add_button_clicked():
    start_frame.pack_forget()
    add_data_container = tk.Frame(container, height=height, width=width)
    add_data_container.pack()
    add_data.control_add_data_container(add_data_container, start_frame)

def view_button_clicked():
    start_frame.pack_forget()
    view_data_container = tk.Frame(container, height=height, width=width)
    view_data_container.pack()
    view_data.control_view_data(view_data_container)

tk.Button(start_frame, text="Add data", command=lambda: add_button_clicked(), font=("Courier", 16)).place(relx=.4,
                                                                                rely=.45,
                                                                                anchor=tk.CENTER)

tk.Button(start_frame, text="View data", command=lambda: view_button_clicked(), font=("Courier", 16)).place(relx=.6,
                                                                                rely=.45,
                                                                                anchor=tk.CENTER)

container.mainloop()


from ee.clickhouse.materialized_columns import materialize
from ee.clickhouse.queries.column_optimizer import ColumnOptimizer
from ee.clickhouse.util import ClickhouseTestMixin
from posthog.models import Action, ActionStep
from posthog.models.filters import Filter
from posthog.test.base import APIBaseTest

PROPERTIES_OF_ALL_TYPES = [
    {"key": "event_prop", "value": ["foo", "bar"], "type": "event"},
    {"key": "person_prop", "value": "efg", "type": "person"},
    {"key": "id", "value": 1, "type": "cohort"},
    {"key": "tag_name", "value": ["label"], "operator": "exact", "type": "element"},
]

BASE_FILTER = Filter({"events": [{"id": "$pageview", "type": "events", "order": 0}]})
FILTER_WITH_PROPERTIES = BASE_FILTER.with_data({"properties": PROPERTIES_OF_ALL_TYPES})

class TestColumnOptimizer(ClickhouseTestMixin, APIBaseTest):
    def setUp(self):
        super().setUp()
        self.team.test_account_filters = PROPERTIES_OF_ALL_TYPES
        self.team.save()

    def test_properties_used_in_filter(self):
        properties_used_in_filter = lambda filter: ColumnOptimizer(filter, self.team.id).properties_used_in_filter

        self.assertEqual(properties_used_in_filter(BASE_FILTER), {})
        self.assertEqual(
            properties_used_in_filter(FILTER_WITH_PROPERTIES),
            {("event_prop", "event"): 1, ("person_prop", "person"): 1, ("id", "cohort"): 1, ("tag_name", "element"): 1},
        )

        # Breakdown cases
        filter = BASE_FILTER.with_data({"breakdown": "some_prop", "breakdown_type": "person"})
        self.assertEqual(properties_used_in_filter(filter), {("some_prop", "person"): 1})

        filter = BASE_FILTER.with_data({"breakdown": "some_prop", "breakdown_type": "event"})
        self.assertEqual(properties_used_in_filter(filter), {("some_prop", "event"): 1})

        filter = BASE_FILTER.with_data({"breakdown": [11], "breakdown_type": "cohort"})
        self.assertEqual(properties_used_in_filter(filter), {})

        # Funnel Correlation cases
        filter = BASE_FILTER.with_data(
            {"funnel_correlation_type": "events", "funnel_correlation_names": ["random_column"]}
        )
        self.assertEqual(properties_used_in_filter(filter), {})

        filter = BASE_FILTER.with_data(
            {"funnel_correlation_type": "properties", "funnel_correlation_names": ["random_column", "$browser"]}
        )
        self.assertEqual(properties_used_in_filter(filter), {("random_column", "person"): 1, ("$browser", "person"): 1})

        filter = BASE_FILTER.with_data({"funnel_correlation_type": "properties"})
        self.assertEqual(properties_used_in_filter(filter), {})

        filter = Filter(
            data={
                "events": [
                    {
                        "id": "$pageview",
                        "type": "events",
                        "order": 0,
                        "math": "sum",
                        "math_property": "numeric_prop",
                        "properties": PROPERTIES_OF_ALL_TYPES,
                    }
                ]
            }
        )
        self.assertEqual(
            properties_used_in_filter(filter),
            {
                ("numeric_prop", "event"): 1,
                ("event_prop", "event"): 1,
                ("person_prop", "person"): 1,
                ("id", "cohort"): 1,
                ("tag_name", "element"): 1,
            },
        )

    def test_properties_used_in_filter_with_actions(self):
        action = Action.objects.create(team=self.team)
        ActionStep.objects.create(
            event="$autocapture", action=action, url="https://example.com/donate", url_matching=ActionStep.EXACT,
        )
        ActionStep.objects.create(
            action=action,
            event="$autocapture",
            tag_name="button",
            text="Pay $10",
            properties=[{"key": "$browser", "value": "Chrome", "type": "person"}],
        )

        filter = Filter(data={"actions": [{"id": action.id, "math": "dau"}]})
        self.assertEqual(
            ColumnOptimizer(filter, self.team.id).properties_used_in_filter,
            {("$current_url", "event"): 1, ("$browser", "person"): 1},
        )

        filter = BASE_FILTER.with_data({"exclusions": [{"id": action.id, "type": "actions"}]})
        self.assertEqual(
            ColumnOptimizer(filter, self.team.id).properties_used_in_filter,
            {("$current_url", "event"): 1, ("$browser", "person"): 1},
        )

    def test_materialized_columns_checks(self):
        optimizer = lambda: ColumnOptimizer(FILTER_WITH_PROPERTIES, self.team.id)

        self.assertEqual(optimizer().event_columns_to_query, {"properties"})
        self.assertEqual(optimizer().person_columns_to_query, {"properties"})

        materialize("events", "event_prop")
        materialize("person", "person_prop")

        self.assertEqual(optimizer().event_columns_to_query, {"mat_event_prop"})
        self.assertEqual(optimizer().person_columns_to_query, {"pmat_person_prop"})

    def test_should_query_element_chain_column(self):
        should_query_elements_chain_column = lambda filter: ColumnOptimizer(
            filter, self.team.id
        ).should_query_elements_chain_column

        self.assertEqual(should_query_elements_chain_column(BASE_FILTER), False)
        self.assertEqual(should_query_elements_chain_column(FILTER_WITH_PROPERTIES), True)

        filter = Filter(
            data={"events": [{"id": "$pageview", "type": "events", "order": 0, "properties": PROPERTIES_OF_ALL_TYPES,}]}
        )
        self.assertEqual(should_query_elements_chain_column(filter), True)

    def test_should_query_element_chain_column_with_actions(self):
        action = Action.objects.create(team=self.team)
        step1 = ActionStep.objects.create(
            event="$autocapture", action=action, url="https://example.com/donate", url_matching=ActionStep.EXACT,
        )

        filter = Filter(data={"actions": [{"id": action.id, "math": "dau"}]})
        self.assertEqual(
            ColumnOptimizer(filter, self.team.id).should_query_elements_chain_column, False,
        )

        ActionStep.objects.create(
            action=action, event="$autocapture", tag_name="button", text="Pay $10",
        )

        self.assertEqual(
            ColumnOptimizer(filter, self.team.id).should_query_elements_chain_column, True,
        )

        filter = BASE_FILTER.with_data({"exclusions": [{"id": action.id, "type": "actions"}]})
        self.assertEqual(
            ColumnOptimizer(filter, self.team.id).should_query_elements_chain_column, True,
        )